diff options
131 files changed, 2521 insertions, 1917 deletions
diff --git a/Doc/c-api/buffer.rst b/Doc/c-api/buffer.rst index 6cb474e..46c19d3 100644 --- a/Doc/c-api/buffer.rst +++ b/Doc/c-api/buffer.rst @@ -133,15 +133,15 @@ a buffer, see :c:func:`PyObject_GetBuffer`. called on non-NULL :c:member:`~Py_buffer.format` values. Important exception: If a consumer requests a buffer without the - :c:macro:`PyBUF_FORMAT` flag, :c:member:`~Py_Buffer.format` will + :c:macro:`PyBUF_FORMAT` flag, :c:member:`~Py_buffer.format` will be set to *NULL*, but :c:member:`~Py_buffer.itemsize` still has the value for the original format. - If :c:member:`~Py_Buffer.shape` is present, the equality + If :c:member:`~Py_buffer.shape` is present, the equality ``product(shape) * itemsize == len`` still holds and the consumer can use :c:member:`~Py_buffer.itemsize` to navigate the buffer. - If :c:member:`~Py_Buffer.shape` is *NULL* as a result of a :c:macro:`PyBUF_SIMPLE` + If :c:member:`~Py_buffer.shape` is *NULL* as a result of a :c:macro:`PyBUF_SIMPLE` or a :c:macro:`PyBUF_WRITABLE` request, the consumer must disregard :c:member:`~Py_buffer.itemsize` and assume ``itemsize == 1``. @@ -156,7 +156,7 @@ a buffer, see :c:func:`PyObject_GetBuffer`. .. c:member:: int ndim The number of dimensions the memory represents as an n-dimensional array. - If it is 0, :c:member:`~Py_Buffer.buf` points to a single item representing + If it is 0, :c:member:`~Py_buffer.buf` points to a single item representing a scalar. In this case, :c:member:`~Py_buffer.shape`, :c:member:`~Py_buffer.strides` and :c:member:`~Py_buffer.suboffsets` MUST be *NULL*. diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst index ed83e82..b5113aa 100644 --- a/Doc/c-api/typeobj.rst +++ b/Doc/c-api/typeobj.rst @@ -94,7 +94,7 @@ type objects) *must* have the :attr:`ob_size` field. This field is not inherited by subtypes. -.. c:member:: char* PyTypeObject.tp_name +.. c:member:: const char* PyTypeObject.tp_name Pointer to a NUL-terminated string containing the name of the type. For types that are accessible as module globals, the string should be the full module @@ -372,7 +372,7 @@ type objects) *must* have the :attr:`ob_size` field. inherited individually. -.. c:member:: long PyTypeObject.tp_flags +.. c:member:: unsigned long PyTypeObject.tp_flags This field is a bit mask of various flags. Some flags indicate variant semantics for certain situations; others are used to indicate that certain @@ -472,7 +472,7 @@ type objects) *must* have the :attr:`ob_size` field. .. versionadded:: 3.4 -.. c:member:: char* PyTypeObject.tp_doc +.. c:member:: const char* PyTypeObject.tp_doc An optional pointer to a NUL-terminated C string giving the docstring for this type object. This is exposed as the :attr:`__doc__` attribute on the type and @@ -619,7 +619,7 @@ type objects) *must* have the :attr:`ob_size` field. +----------------+------------+ -.. c:member:: long PyTypeObject.tp_weaklistoffset +.. c:member:: Py_ssize_t PyTypeObject.tp_weaklistoffset If the instances of this type are weakly referenceable, this field is greater than zero and contains the offset in the instance structure of the weak @@ -786,7 +786,7 @@ type objects) *must* have the :attr:`ob_size` field. .. XXX explain. -.. c:member:: long PyTypeObject.tp_dictoffset +.. c:member:: Py_ssize_t PyTypeObject.tp_dictoffset If the instances of this type have a dictionary containing instance variables, this field is non-zero and contains the offset in the instances of the type of diff --git a/Doc/extending/newtypes.rst b/Doc/extending/newtypes.rst index b7e35f4..f60e208 100644 --- a/Doc/extending/newtypes.rst +++ b/Doc/extending/newtypes.rst @@ -893,20 +893,20 @@ fields in the right order! It's often easiest to find an example that includes all the fields you need (even if they're initialized to ``0``) and then change the values to suit your new type. :: - char *tp_name; /* For printing */ + const char *tp_name; /* For printing */ The name of the type - as mentioned in the last section, this will appear in various places, almost entirely for diagnostic purposes. Try to choose something that will be helpful in such a situation! :: - int tp_basicsize, tp_itemsize; /* For allocation */ + Py_ssize_t tp_basicsize, tp_itemsize; /* For allocation */ These fields tell the runtime how much memory to allocate when new objects of this type are created. Python has some built-in support for variable length structures (think: strings, lists) which is where the :c:member:`~PyTypeObject.tp_itemsize` field comes in. This will be dealt with later. :: - char *tp_doc; + const char *tp_doc; Here you can put a string (or its address) that you want returned when the Python script references ``obj.__doc__`` to retrieve the doc string. diff --git a/Doc/extending/windows.rst b/Doc/extending/windows.rst index f0c69b8..67bdd47 100644 --- a/Doc/extending/windows.rst +++ b/Doc/extending/windows.rst @@ -37,149 +37,9 @@ There are two approaches to building extension modules on Windows, just as there are on Unix: use the :mod:`distutils` package to control the build process, or do things manually. The distutils approach works well for most extensions; documentation on using :mod:`distutils` to build and package extension modules -is available in :ref:`distutils-index`. This section describes the manual -approach to building Python extensions written in C or C++. - -To build extensions using these instructions, you need to have a copy of the -Python sources of the same version as your installed Python. You will need -Microsoft Visual C++ "Developer Studio"; project files are supplied for VC++ -version 7.1, but you can use older versions of VC++. Notice that you should use -the same version of VC++that was used to build Python itself. The example files -described here are distributed with the Python sources in the -:file:`PC\\example_nt\\` directory. - -#. **Copy the example files** --- The :file:`example_nt` directory is a - subdirectory of the :file:`PC` directory, in order to keep all the PC-specific - files under the same directory in the source distribution. However, the - :file:`example_nt` directory can't actually be used from this location. You - first need to copy or move it up one level, so that :file:`example_nt` is a - sibling of the :file:`PC` and :file:`Include` directories. Do all your work - from within this new location. - -#. **Open the project** --- From VC++, use the :menuselection:`File --> Open - Solution` dialog (not :menuselection:`File --> Open`!). Navigate to and select - the file :file:`example.sln`, in the *copy* of the :file:`example_nt` directory - you made above. Click Open. - -#. **Build the example DLL** --- In order to check that everything is set up - right, try building: - -#. Select a configuration. This step is optional. Choose - :menuselection:`Build --> Configuration Manager --> Active Solution Configuration` - and select either :guilabel:`Release` or :guilabel:`Debug`. If you skip this - step, VC++ will use the Debug configuration by default. - -#. Build the DLL. Choose :menuselection:`Build --> Build Solution`. This - creates all intermediate and result files in a subdirectory called either - :file:`Debug` or :file:`Release`, depending on which configuration you selected - in the preceding step. - -#. **Testing the debug-mode DLL** --- Once the Debug build has succeeded, bring - up a DOS box, and change to the :file:`example_nt\\Debug` directory. You should - now be able to repeat the following session (``C>`` is the DOS prompt, ``>>>`` - is the Python prompt; note that build information and various debug output from - Python may not match this screen dump exactly):: - - C>..\..\PCbuild\python_d - Adding parser accelerators ... - Done. - Python 2.2 (#28, Dec 19 2001, 23:26:37) [MSC 32 bit (Intel)] on win32 - Type "copyright", "credits" or "license" for more information. - >>> import example - [4897 refs] - >>> example.foo() - Hello, world - [4903 refs] - >>> - - Congratulations! You've successfully built your first Python extension module. - -#. **Creating your own project** --- Choose a name and create a directory for - it. Copy your C sources into it. Note that the module source file name does - not necessarily have to match the module name, but the name of the - initialization function should match the module name --- you can only import a - module :mod:`spam` if its initialization function is called :c:func:`PyInit_spam`, - (see :ref:`building`, or use the minimal :file:`Modules/xxmodule.c` as a guide). - By convention, it lives in a file called :file:`spam.c` or :file:`spammodule.c`. - The output file should be called :file:`spam.pyd` (in Release mode) or - :file:`spam_d.pyd` (in Debug mode). The extension :file:`.pyd` was chosen - to avoid confusion with a system library :file:`spam.dll` to which your module - could be a Python interface. - - Now your options are: - -#. Copy :file:`example.sln` and :file:`example.vcproj`, rename them to - :file:`spam.\*`, and edit them by hand, or - -#. Create a brand new project; instructions are below. - - In either case, copy :file:`example_nt\\example.def` to :file:`spam\\spam.def`, - and edit the new :file:`spam.def` so its second line contains the string - '``initspam``'. If you created a new project yourself, add the file - :file:`spam.def` to the project now. (This is an annoying little file with only - two lines. An alternative approach is to forget about the :file:`.def` file, - and add the option :option:`/export:initspam` somewhere to the Link settings, by - manually editing the setting in Project Properties dialog). - -#. **Creating a brand new project** --- Use the :menuselection:`File --> New - --> Project` dialog to create a new Project Workspace. Select :guilabel:`Visual - C++ Projects/Win32/ Win32 Project`, enter the name (``spam``), and make sure the - Location is set to parent of the :file:`spam` directory you have created (which - should be a direct subdirectory of the Python build tree, a sibling of - :file:`Include` and :file:`PC`). Select Win32 as the platform (in my version, - this is the only choice). Make sure the Create new workspace radio button is - selected. Click OK. - - You should now create the file :file:`spam.def` as instructed in the previous - section. Add the source files to the project, using :menuselection:`Project --> - Add Existing Item`. Set the pattern to ``*.*`` and select both :file:`spam.c` - and :file:`spam.def` and click OK. (Inserting them one by one is fine too.) - - Now open the :menuselection:`Project --> spam properties` dialog. You only need - to change a few settings. Make sure :guilabel:`All Configurations` is selected - from the :guilabel:`Settings for:` dropdown list. Select the C/C++ tab. Choose - the General category in the popup menu at the top. Type the following text in - the entry box labeled :guilabel:`Additional Include Directories`:: - - ..\Include,..\PC - - Then, choose the General category in the Linker tab, and enter :: - - ..\PCbuild - - in the text box labelled :guilabel:`Additional library Directories`. - - Now you need to add some mode-specific settings: - - Select :guilabel:`Release` in the :guilabel:`Configuration` dropdown list. - Choose the :guilabel:`Link` tab, choose the :guilabel:`Input` category, and - append ``pythonXY.lib`` to the list in the :guilabel:`Additional Dependencies` - box. - - Select :guilabel:`Debug` in the :guilabel:`Configuration` dropdown list, and - append ``pythonXY_d.lib`` to the list in the :guilabel:`Additional Dependencies` - box. Then click the C/C++ tab, select :guilabel:`Code Generation`, and select - :guilabel:`Multi-threaded Debug DLL` from the :guilabel:`Runtime library` - dropdown list. - - Select :guilabel:`Release` again from the :guilabel:`Configuration` dropdown - list. Select :guilabel:`Multi-threaded DLL` from the :guilabel:`Runtime - library` dropdown list. - -If your module creates a new type, you may have trouble with this line:: - - PyVarObject_HEAD_INIT(&PyType_Type, 0) - -Static type object initializers in extension modules may cause -compiles to fail with an error message like "initializer not a -constant". This shows up when building DLL under MSVC. Change it to:: - - PyVarObject_HEAD_INIT(NULL, 0) - -and add the following to the module initialization function:: - - if (PyType_Ready(&MyObject_Type) < 0) - return NULL; +is available in :ref:`distutils-index`. If you find you really need to do +things manually, it may be instructive to study the project file for the +:source:`winsound <PCbuild/winsound.vcxproj>` standard library module. .. _dynamic-linking: diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst index 2d3cb1c..67a9c56 100644 --- a/Doc/faq/programming.rst +++ b/Doc/faq/programming.rst @@ -1164,6 +1164,8 @@ analogue of lisp car is ``lisp_list[0]`` and the analogue of cdr is usually a lot slower than using Python lists. +.. _faq-multidimensional-list: + How do I create a multidimensional list? ---------------------------------------- diff --git a/Doc/includes/typestruct.h b/Doc/includes/typestruct.h index 5c9f8f5..9f47899 100644 --- a/Doc/includes/typestruct.h +++ b/Doc/includes/typestruct.h @@ -1,7 +1,7 @@ typedef struct _typeobject { PyObject_VAR_HEAD - char *tp_name; /* For printing, in format "<module>.<name>" */ - int tp_basicsize, tp_itemsize; /* For allocation */ + const char *tp_name; /* For printing, in format "<module>.<name>" */ + Py_ssize_t tp_basicsize, tp_itemsize; /* For allocation */ /* Methods to implement standard operations */ @@ -9,7 +9,8 @@ typedef struct _typeobject { printfunc tp_print; getattrfunc tp_getattr; setattrfunc tp_setattr; - PyAsyncMethods *tp_as_async; + PyAsyncMethods *tp_as_async; /* formerly known as tp_compare (Python 2) + or tp_reserved (Python 3) */ reprfunc tp_repr; /* Method suites for standard classes */ @@ -30,9 +31,9 @@ typedef struct _typeobject { PyBufferProcs *tp_as_buffer; /* Flags to define presence of optional/expanded features */ - long tp_flags; + unsigned long tp_flags; - char *tp_doc; /* Documentation string */ + const char *tp_doc; /* Documentation string */ /* call function for all accessible objects */ traverseproc tp_traverse; @@ -44,7 +45,7 @@ typedef struct _typeobject { richcmpfunc tp_richcompare; /* weak reference enabler */ - long tp_weaklistoffset; + Py_ssize_t tp_weaklistoffset; /* Iterators */ getiterfunc tp_iter; @@ -58,7 +59,7 @@ typedef struct _typeobject { PyObject *tp_dict; descrgetfunc tp_descr_get; descrsetfunc tp_descr_set; - long tp_dictoffset; + Py_ssize_t tp_dictoffset; initproc tp_init; allocfunc tp_alloc; newfunc tp_new; @@ -69,7 +70,6 @@ typedef struct _typeobject { PyObject *tp_cache; PyObject *tp_subclasses; PyObject *tp_weaklist; - destructor tp_del; /* Type attribute cache version tag. Added in version 2.6 */ diff --git a/Doc/library/_thread.rst b/Doc/library/_thread.rst index a787e2f..7122861 100644 --- a/Doc/library/_thread.rst +++ b/Doc/library/_thread.rst @@ -93,7 +93,8 @@ It defines the following constants and functions: Return the thread stack size used when creating new threads. The optional *size* argument specifies the stack size to be used for subsequently created threads, and must be 0 (use platform or configured default) or a positive - integer value of at least 32,768 (32 KiB). If changing the thread stack size is + integer value of at least 32,768 (32 KiB). If *size* is not specified, + 0 is used. If changing the thread stack size is unsupported, a :exc:`RuntimeError` is raised. If the specified stack size is invalid, a :exc:`ValueError` is raised and the stack size is unmodified. 32 KiB is currently the minimum supported stack size value to guarantee sufficient diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst index 7d7caa8..146c8c9 100644 --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -6,7 +6,7 @@ Base Event Loop =============== The event loop is the central execution device provided by :mod:`asyncio`. -It provides multiple facilities, amongst which: +It provides multiple facilities, including: * Registering, executing and cancelling delayed calls (timeouts). diff --git a/Doc/library/asyncio-subprocess.rst b/Doc/library/asyncio-subprocess.rst index 3c9e3cb..c0704cd 100644 --- a/Doc/library/asyncio-subprocess.rst +++ b/Doc/library/asyncio-subprocess.rst @@ -303,7 +303,7 @@ Process .. _asyncio-subprocess-threads: Subprocess and threads -====================== +---------------------- asyncio supports running subprocesses from different threads, but there are limits: @@ -322,10 +322,10 @@ The :class:`asyncio.subprocess.Process` class is not thread safe. Subprocess examples -=================== +------------------- Subprocess using transport and protocol ---------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Example of a subprocess protocol using to get the output of a subprocess and to wait for the subprocess exit. The subprocess is created by the @@ -381,7 +381,7 @@ wait for the subprocess exit. The subprocess is created by the Subprocess using streams ------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^ Example using the :class:`~asyncio.subprocess.Process` class to control the subprocess and the :class:`StreamReader` class to read from the standard diff --git a/Doc/library/asyncio.rst b/Doc/library/asyncio.rst index b292ae9..9b4d65e 100644 --- a/Doc/library/asyncio.rst +++ b/Doc/library/asyncio.rst @@ -4,6 +4,13 @@ .. module:: asyncio :synopsis: Asynchronous I/O, event loop, coroutines and tasks. +.. note:: + + The asyncio package has been included in the standard library on a + :term:`provisional basis <provisional package>`. Backwards incompatible + changes (up to and including removal of the module) may occur if deemed + necessary by the core developers. + .. versionadded:: 3.4 **Source code:** :source:`Lib/asyncio/` diff --git a/Doc/library/binascii.rst b/Doc/library/binascii.rst index 3f7df74..e3f134b 100644 --- a/Doc/library/binascii.rst +++ b/Doc/library/binascii.rst @@ -59,7 +59,7 @@ The :mod:`binascii` module defines the following functions: should be at most 57 to adhere to the base64 standard. -.. function:: a2b_qp(string, header=False) +.. function:: a2b_qp(data, header=False) Convert a block of quoted-printable data back to binary and return the binary data. More than one line may be passed at a time. If the optional argument diff --git a/Doc/library/collections.rst b/Doc/library/collections.rst index f2befe7..8121cc4 100644 --- a/Doc/library/collections.rst +++ b/Doc/library/collections.rst @@ -842,10 +842,10 @@ field names, the method and attribute names start with an underscore. .. method:: somenamedtuple._asdict() Return a new :class:`OrderedDict` which maps field names to their corresponding - values. Note, this method is no longer needed now that the same effect can - be achieved by using the built-in :func:`vars` function:: + values:: - >>> vars(p) + >>> p = Point(x=11, y=22) + >>> p._asdict() OrderedDict([('x', 11), ('y', 22)]) .. versionchanged:: 3.1 @@ -1157,3 +1157,7 @@ attribute. be an instance of :class:`bytes`, :class:`str`, :class:`UserString` (or a subclass) or an arbitrary sequence which can be converted into a string using the built-in :func:`str` function. + + .. versionchanged:: 3.5 + New methods ``__getnewargs__``, ``__rmod__``, ``casefold``, + ``format_map``, ``isprintable``, and ``maketrans``. diff --git a/Doc/library/compileall.rst b/Doc/library/compileall.rst index 0325f1a..82f3753 100644 --- a/Doc/library/compileall.rst +++ b/Doc/library/compileall.rst @@ -88,14 +88,9 @@ compile Python sources. Added the ``-i``, ``-b`` and ``-h`` options. .. versionchanged:: 3.5 - Added the ``-j`` and ``-r`` options. - -.. versionchanged:: 3.5 - ``-q`` option was changed to a multilevel value. - -.. versionchanged:: 3.5 - ``-b`` will always produce a byte-code file ending in ``.pyc``, never - ``.pyo``. + Added the ``-j``, ``-r``, and ``-qq`` options. ``-q`` option + was changed to a multilevel value. ``b`` will always produce a + byte-code file ending in ``.pyc``, never ``.pyo``. There is no command-line option to control the optimization level used by the diff --git a/Doc/library/csv.rst b/Doc/library/csv.rst index 602e909..4fcfaef 100644 --- a/Doc/library/csv.rst +++ b/Doc/library/csv.rst @@ -325,7 +325,7 @@ Dialects support the following attributes: .. attribute:: Dialect.doublequote - Controls how instances of *quotechar* appearing inside a field should be + Controls how instances of *quotechar* appearing inside a field should themselves be quoted. When :const:`True`, the character is doubled. When :const:`False`, the *escapechar* is used as a prefix to the *quotechar*. It defaults to :const:`True`. diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst index 7a214ed..1bcb3a4 100644 --- a/Doc/library/dis.rst +++ b/Doc/library/dis.rst @@ -946,8 +946,8 @@ the more significant byte last. Creates a new function object, sets its *__closure__* slot, and pushes it on the stack. TOS is the :term:`qualified name` of the function, TOS1 is the code associated with the function, and TOS2 is the tuple containing cells for - the closure's free variables. The function also has *argc* default - parameters, which are found below the cells. + the closure's free variables. *argc* is interpreted as in ``MAKE_FUNCTION``; + the annotations and defaults are also in the same order below TOS2. .. opcode:: BUILD_SLICE (argc) diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst index d3b838c..9b4f9b4 100644 --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -466,7 +466,7 @@ The complete signature is:: :type: type to mix in to new Enum class. -:start: number to start counting at if only names are passed in +:start: number to start counting at if only names are passed in. .. versionchanged:: 3.5 The *start* parameter was added. diff --git a/Doc/library/html.parser.rst b/Doc/library/html.parser.rst index b84c60b..824995e 100644 --- a/Doc/library/html.parser.rst +++ b/Doc/library/html.parser.rst @@ -185,7 +185,7 @@ implementations do nothing (except for :meth:`~HTMLParser.handle_startendtag`): The content of Internet Explorer conditional comments (condcoms) will also be sent to this method, so, for ``<!--[if IE 9]>IE9-specific content<![endif]-->``, - this method will receive ``'[if IE 9]>IE-specific content<![endif]'``. + this method will receive ``'[if IE 9]>IE9-specific content<![endif]'``. .. method:: HTMLParser.handle_decl(decl) diff --git a/Doc/library/imghdr.rst b/Doc/library/imghdr.rst index c60df24..f11f6dc 100644 --- a/Doc/library/imghdr.rst +++ b/Doc/library/imghdr.rst @@ -54,10 +54,8 @@ from :func:`what`: +------------+-----------------------------------+ .. versionadded:: 3.5 - The *exr* format was added. + The *exr* and *webp* formats were added. -.. versionchanged:: 3.5 - The *webp* type was added. You can extend the list of file types :mod:`imghdr` can recognize by appending to this variable: diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst index 3b62e7f..26a2913 100644 --- a/Doc/library/inspect.rst +++ b/Doc/library/inspect.rst @@ -178,6 +178,10 @@ attributes: +-----------+-----------------+---------------------------+ | | gi_code | code | +-----------+-----------------+---------------------------+ +| | gi_yieldfrom | object being iterated by | +| | | ``yield from``, or | +| | | ``None`` | ++-----------+-----------------+---------------------------+ | coroutine | __name__ | name | +-----------+-----------------+---------------------------+ | | __qualname__ | qualified name | @@ -191,10 +195,6 @@ attributes: +-----------+-----------------+---------------------------+ | | cr_code | code | +-----------+-----------------+---------------------------+ -| | gi_yieldfrom | object being iterated by | -| | | ``yield from``, or | -| | | ``None`` | -+-----------+-----------------+---------------------------+ | builtin | __doc__ | documentation string | +-----------+-----------------+---------------------------+ | | __name__ | original name of this | @@ -209,9 +209,10 @@ attributes: .. versionchanged:: 3.5 - Add ``__qualname__`` attribute to generators. The ``__name__`` attribute of - generators is now set from the function name, instead of the code name, and - it can now be modified. + Add ``__qualname__`` and ``gi_yieldfrom`` attributes to generators. + + The ``__name__`` attribute of generators is now set from the function + name, instead of the code name, and it can now be modified. .. function:: getmembers(object[, predicate]) @@ -1038,6 +1039,11 @@ line. returned list represents *frame*; the last entry represents the outermost call on *frame*'s stack. + .. versionchanged:: 3.5 + A list of :term:`named tuples <named tuple>` + ``FrameInfo(frame, filename, lineno, function, code_context, index)`` + is returned. + .. function:: getinnerframes(traceback, context=1) @@ -1046,6 +1052,11 @@ line. list represents *traceback*; the last entry represents where the exception was raised. + .. versionchanged:: 3.5 + A list of :term:`named tuples <named tuple>` + ``FrameInfo(frame, filename, lineno, function, code_context, index)`` + is returned. + .. function:: currentframe() @@ -1065,6 +1076,11 @@ line. returned list represents the caller; the last entry represents the outermost call on the stack. + .. versionchanged:: 3.5 + A list of :term:`named tuples <named tuple>` + ``FrameInfo(frame, filename, lineno, function, code_context, index)`` + is returned. + .. function:: trace(context=1) @@ -1073,6 +1089,11 @@ line. entry in the list represents the caller; the last entry represents where the exception was raised. + .. versionchanged:: 3.5 + A list of :term:`named tuples <named tuple>` + ``FrameInfo(frame, filename, lineno, function, code_context, index)`` + is returned. + Fetching attributes statically ------------------------------ diff --git a/Doc/library/io.rst b/Doc/library/io.rst index f009c65..48fd226 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -301,11 +301,11 @@ I/O Base Classes Note that it's already possible to iterate on file objects using ``for line in file: ...`` without calling ``file.readlines()``. - .. method:: seek(offset, whence=SEEK_SET) + .. method:: seek(offset[, whence]) Change the stream position to the given byte *offset*. *offset* is - interpreted relative to the position indicated by *whence*. Values for - *whence* are: + interpreted relative to the position indicated by *whence*. The default + value for *whence* is :data:`SEEK_SET`. Values for *whence* are: * :data:`SEEK_SET` or ``0`` -- start of the stream (the default); *offset* should be zero or positive @@ -487,7 +487,7 @@ I/O Base Classes .. method:: readinto1(b) - Read up to ``len(b)`` bytes into bytearray *b*, ,using at most one call to + Read up to ``len(b)`` bytes into bytearray *b*, using at most one call to the underlying raw stream's :meth:`~RawIOBase.read` (or :meth:`~RawIOBase.readinto`) method. Return the number of bytes read. @@ -783,10 +783,11 @@ Text I/O If *size* is specified, at most *size* characters will be read. - .. method:: seek(offset, whence=SEEK_SET) + .. method:: seek(offset[, whence]) - Change the stream position to the given *offset*. Behaviour depends - on the *whence* parameter: + Change the stream position to the given *offset*. Behaviour depends on + the *whence* parameter. The default value for *whence* is + :data:`SEEK_SET`. * :data:`SEEK_SET` or ``0``: seek from the start of the stream (the default); *offset* must either be a number returned by diff --git a/Doc/library/ipaddress.rst b/Doc/library/ipaddress.rst index 7b59440..90fcc74 100644 --- a/Doc/library/ipaddress.rst +++ b/Doc/library/ipaddress.rst @@ -575,7 +575,7 @@ so to avoid duplication they are only documented for :class:`IPv4Network`. single-address network, with the network address being *address* and the mask being ``/128``. - 3. An integer packed into a :class:`bytes` object of length 16, bit-endian. + 3. An integer packed into a :class:`bytes` object of length 16, big-endian. The interpretation is similar to an integer *address*. 4. A two-tuple of an address description and a netmask, where the address diff --git a/Doc/library/os.path.rst b/Doc/library/os.path.rst index e4fe44e..a3fe73c 100644 --- a/Doc/library/os.path.rst +++ b/Doc/library/os.path.rst @@ -82,8 +82,21 @@ the :mod:`glob` module.) Return the longest path prefix (taken character-by-character) that is a prefix of all paths in *list*. If *list* is empty, return the empty string - (``''``). Note that this may return invalid paths because it works a - character at a time. To obtain a valid path, see :func:`commonpath`. + (``''``). + + .. note:: + + This function may return invalid paths because it works a + character at a time. To obtain a valid path, see + :func:`commonpath`. + + :: + + >>> os.path.commonprefix(['/usr/lib', '/usr/local/lib']) + '/usr/l' + + >>> os.path.commonpath(['/usr/lib', '/usr/local/lib']) + '/usr' .. function:: dirname(path) diff --git a/Doc/library/os.rst b/Doc/library/os.rst index d4032e0..13f83c6 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -857,9 +857,9 @@ as internal buffering of data. :data:`os.SEEK_HOLE` or :data:`os.SEEK_DATA`. -.. function:: open(file, flags, mode=0o777, *, dir_fd=None) +.. function:: open(path, flags, mode=0o777, *, dir_fd=None) - Open the file *file* and set various flags according to *flags* and possibly + Open the file *path* and set various flags according to *flags* and possibly its mode according to *mode*. When computing *mode*, the current umask value is first masked out. Return the file descriptor for the newly opened file. The new file descriptor is :ref:`non-inheritable <fd_inheritance>`. @@ -1071,10 +1071,10 @@ or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Window :exc:`InterruptedError` exception (see :pep:`475` for the rationale). -.. function:: sendfile(out, in, offset, nbytes) - sendfile(out, in, offset, nbytes, headers=None, trailers=None, flags=0) +.. function:: sendfile(out, in, offset, count) + sendfile(out, in, offset, count, [headers], [trailers], flags=0) - Copy *nbytes* bytes from file descriptor *in* to file descriptor *out* + Copy *count* bytes from file descriptor *in* to file descriptor *out* starting at *offset*. Return the number of bytes sent. When EOF is reached return 0. @@ -1088,7 +1088,7 @@ or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Window *trailers* are arbitrary sequences of buffers that are written before and after the data from *in* is written. It returns the same as the first case. - On Mac OS X and FreeBSD, a value of 0 for *nbytes* specifies to send until + On Mac OS X and FreeBSD, a value of 0 for *count* specifies to send until the end of *in* is reached. All platforms support sockets as *out* file descriptor, and some platforms @@ -1690,10 +1690,10 @@ features: The *dir_fd* argument. -.. function:: mknod(filename, mode=0o600, device=0, *, dir_fd=None) +.. function:: mknod(path, mode=0o600, device=0, *, dir_fd=None) Create a filesystem node (file, device special file or named pipe) named - *filename*. *mode* specifies both the permissions to use and the type of node + *path*. *mode* specifies both the permissions to use and the type of node to be created, being combined (bitwise OR) with one of ``stat.S_IFREG``, ``stat.S_IFCHR``, ``stat.S_IFBLK``, and ``stat.S_IFIFO`` (those constants are available in :mod:`stat`). For ``stat.S_IFCHR`` and ``stat.S_IFBLK``, @@ -2389,9 +2389,9 @@ features: .. versionadded:: 3.3 -.. function:: symlink(source, link_name, target_is_directory=False, *, dir_fd=None) +.. function:: symlink(src, dst, target_is_directory=False, *, dir_fd=None) - Create a symbolic link pointing to *source* named *link_name*. + Create a symbolic link pointing to *src* named *dst*. On Windows, a symlink represents either a file or a directory, and does not morph to the target dynamically. If the target is present, the type of the @@ -2461,20 +2461,20 @@ features: The *dir_fd* parameter. -.. function:: utime(path, times=None, *, ns=None, dir_fd=None, follow_symlinks=True) +.. function:: utime(path, times=None, *[, ns], dir_fd=None, follow_symlinks=True) Set the access and modified times of the file specified by *path*. :func:`utime` takes two optional parameters, *times* and *ns*. These specify the times set on *path* and are used as follows: - - If *ns* is not ``None``, + - If *ns* is specified, it must be a 2-tuple of the form ``(atime_ns, mtime_ns)`` where each member is an int expressing nanoseconds. - If *times* is not ``None``, it must be a 2-tuple of the form ``(atime, mtime)`` where each member is an int or float expressing seconds. - - If *times* and *ns* are both ``None``, + - If *times* is ``None`` and *ns* is unspecified, this is equivalent to specifying ``ns=(atime_ns, mtime_ns)`` where both times are the current time. @@ -3023,9 +3023,10 @@ written in Python, such as a mail server's external command delivery program. Availability: Unix. -.. function:: popen(command, mode='r', buffering=-1) +.. function:: popen(cmd, mode='r', buffering=-1) - Open a pipe to or from *command*. The return value is an open file object + Open a pipe to or from command *cmd*. + The return value is an open file object connected to the pipe, which can be read or written depending on whether *mode* is ``'r'`` (default) or ``'w'``. The *buffering* argument has the same meaning as the corresponding argument to the built-in :func:`open` function. The diff --git a/Doc/library/profile.rst b/Doc/library/profile.rst index 2928821..959d9b9 100644 --- a/Doc/library/profile.rst +++ b/Doc/library/profile.rst @@ -33,7 +33,7 @@ profiling interface: 2. :mod:`profile`, a pure Python module whose interface is imitated by :mod:`cProfile`, but which adds significant overhead to profiled programs. If you're trying to extend the profiler in some way, the task might be easier - with this module. + with this module. Originally designed and written by Jim Roskind. .. note:: diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst index 1dcdb2f..131100a 100644 --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -106,8 +106,30 @@ created. Socket addresses are represented as follows: .. versionadded:: 3.3 -- Certain other address families (:const:`AF_BLUETOOTH`, :const:`AF_PACKET`, - :const:`AF_CAN`) support specific representations. +- :const:`AF_BLUETOOTH` supports the following protocols and address + formats: + + - :const:`BTPROTO_L2CAP` accepts ``(bdaddr, psm)`` where ``bdaddr`` is + the Bluetooth address as a string and ``psm`` is an integer. + + - :const:`BTPROTO_RFCOMM` accepts ``(bdaddr, channel)`` where ``bdaddr`` + is the Bluetooth address as a string and ``channel`` is an integer. + + - :const:`BTPROTO_HCI` accepts ``(device_id,)`` where ``device_id`` is + either an integer or a string with the Bluetooth address of the + interface. (This depends on your OS; NetBSD and DragonFlyBSD expect + a Bluetooth address while everything else expects an integer.) + + .. versionchanged:: 3.2 + NetBSD and DragonFlyBSD support added. + + - :const:`BTPROTO_SCO` accepts ``bdaddr`` where ``bdaddr`` is a + :class:`bytes` object containing the Bluetooth address in a + string format. (ex. ``b'12:23:34:45:56:67'``) This protocol is not + supported under FreeBSD. + +- Certain other address families (:const:`AF_PACKET`, :const:`AF_CAN`) + support specific representations. .. XXX document them! diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index 8573416..b3c9e23 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -854,8 +854,8 @@ operations have the same priority as the corresponding numeric operations. | ``s + t`` | the concatenation of *s* and | (6)(7) | | | *t* | | +--------------------------+--------------------------------+----------+ -| ``s * n`` or | *n* shallow copies of *s* | (2)(7) | -| ``n * s`` | concatenated | | +| ``s * n`` or | equivalent to adding *s* to | (2)(7) | +| ``n * s`` | itself *n* times | | +--------------------------+--------------------------------+----------+ | ``s[i]`` | *i*\ th item of *s*, origin 0 | \(3) | +--------------------------+--------------------------------+----------+ @@ -897,9 +897,9 @@ Notes: (2) Values of *n* less than ``0`` are treated as ``0`` (which yields an empty - sequence of the same type as *s*). Note also that the copies are shallow; - nested structures are not copied. This often haunts new Python programmers; - consider:: + sequence of the same type as *s*). Note that items in the sequence *s* + are not copied; they are referenced multiple times. This often haunts + new Python programmers; consider:: >>> lists = [[]] * 3 >>> lists @@ -909,7 +909,7 @@ Notes: [[3], [3], [3]] What has happened is that ``[[]]`` is a one-element list containing an empty - list, so all three elements of ``[[]] * 3`` are (pointers to) this single empty + list, so all three elements of ``[[]] * 3`` are references to this single empty list. Modifying any of the elements of ``lists`` modifies this single list. You can create a list of different lists this way:: @@ -920,6 +920,9 @@ Notes: >>> lists [[3], [5], [7]] + Further explanation is available in the FAQ entry + :ref:`faq-multidimensional-list`. + (3) If *i* or *j* is negative, the index is relative to the end of the string: ``len(s) + i`` or ``len(s) + j`` is substituted. But note that ``-0`` is diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst index efcacce..adf99ec 100644 --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -1068,7 +1068,7 @@ Return code handling translates as follows:: if rc is not None and rc >> 8: print("There were some errors") ==> - process = Popen(cmd, 'w', stdin=PIPE) + process = Popen(cmd, stdin=PIPE) ... process.stdin.close() if process.wait() != 0: diff --git a/Doc/library/tempfile.rst b/Doc/library/tempfile.rst index 0387fb1..83f9941 100644 --- a/Doc/library/tempfile.rst +++ b/Doc/library/tempfile.rst @@ -16,16 +16,18 @@ -------------- -This module generates temporary files and directories. It works on all -supported platforms. It provides three new functions, -:func:`NamedTemporaryFile`, :func:`mkstemp`, and :func:`mkdtemp`, which should -eliminate all remaining need to use the insecure :func:`mktemp` function. -Temporary file names created by this module no longer contain the process ID; -instead a string of six random characters is used. - -Also, all the user-callable functions now take additional arguments which -allow direct control over the location and name of temporary files. It is -no longer necessary to use the global *tempdir* variable. +This module creates temporary files and directories. It works on all +supported platforms. :class:`TemporaryFile`, :class:`NamedTemporaryFile`, +:class:`TemporaryDirectory`, and :class:`SpooledTemporaryFile` are high-level +interfaces which provide automatic cleanup and can be used as +context managers. :func:`mkstemp` and +:func:`mkdtemp` are lower-level functions which require manual cleanup. + +All the user-callable functions and constructors take additional arguments which +allow direct control over the location and name of temporary files and +directories. Files names used by this module include a string of +random characters which allows those files to be securely created in +shared temporary directories. To maintain backward compatibility, the argument order is somewhat odd; it is recommended to use keyword arguments for clarity. @@ -34,28 +36,33 @@ The module defines the following user-callable items: .. function:: TemporaryFile(mode='w+b', buffering=None, encoding=None, newline=None, suffix='', prefix='tmp', dir=None) Return a :term:`file-like object` that can be used as a temporary storage area. - The file is created using :func:`mkstemp`. It will be destroyed as soon + The file is created securely, using the same rules as :func:`mkstemp`. It will be destroyed as soon as it is closed (including an implicit close when the object is garbage - collected). Under Unix, the directory entry for the file is removed + collected). Under Unix, the directory entry for the file is either not created at all or is removed immediately after the file is created. Other platforms do not support this; your code should not rely on a temporary file created using this function having or not having a visible name in the file system. + The resulting object can be used as a context manager (see + :ref:`tempfile-examples`). On completion of the context or + destruction of the file object the temporary file will be removed + from the filesystem. + The *mode* parameter defaults to ``'w+b'`` so that the file created can be read and written without being closed. Binary mode is used so that it behaves consistently on all platforms without regard for the data that is stored. *buffering*, *encoding* and *newline* are interpreted as for :func:`open`. - The *dir*, *prefix* and *suffix* parameters are passed to :func:`mkstemp`. + The *dir*, *prefix* and *suffix* parameters have the same meaning + as with :func:`mkstemp`. The returned object is a true file object on POSIX platforms. On other platforms, it is a file-like object whose :attr:`!file` attribute is the - underlying true file object. This file-like object can be used in a - :keyword:`with` statement, just like a normal file. + underlying true file object. The :py:data:`os.O_TMPFILE` flag is used if it is available and works - (Linux-specific, require Linux kernel 3.11 or later). + (Linux-specific, requires Linux kernel 3.11 or later). .. versionchanged:: 3.5 @@ -101,10 +108,9 @@ The module defines the following user-callable items: .. function:: TemporaryDirectory(suffix='', prefix='tmp', dir=None) - This function creates a temporary directory using :func:`mkdtemp` - (the supplied arguments are passed directly to the underlying function). + This function securely creates a temporary directory using the same rules as :func:`mkdtemp`. The resulting object can be used as a context manager (see - :ref:`context-managers`). On completion of the context or destruction + :ref:`tempfile-examples`). On completion of the context or destruction of the temporary directory object the newly created temporary directory and all its contents are removed from the filesystem. @@ -194,49 +200,14 @@ The module defines the following user-callable items: an appropriate default value to be used. -.. function:: mktemp(suffix='', prefix='tmp', dir=None) - - .. deprecated:: 2.3 - Use :func:`mkstemp` instead. - - Return an absolute pathname of a file that did not exist at the time the - call is made. The *prefix*, *suffix*, and *dir* arguments are the same - as for :func:`mkstemp`. - - .. warning:: - - Use of this function may introduce a security hole in your program. By - the time you get around to doing anything with the file name it returns, - someone else may have beaten you to the punch. :func:`mktemp` usage can - be replaced easily with :func:`NamedTemporaryFile`, passing it the - ``delete=False`` parameter:: - - >>> f = NamedTemporaryFile(delete=False) - >>> f.name - '/tmp/tmptjujjt' - >>> f.write(b"Hello World!\n") - 13 - >>> f.close() - >>> os.unlink(f.name) - >>> os.path.exists(f.name) - False - -The module uses a global variable that tell it how to construct a -temporary name. They are initialized at the first call to any of the -functions above. The caller may change them, but this is discouraged; use -the appropriate function arguments, instead. - +.. function:: gettempdir() -.. data:: tempdir + Return the name of the directory used for temporary files. This + defines the default value for the *dir* argument to all functions + in this module. - When set to a value other than ``None``, this variable defines the - default value for the *dir* argument to all the functions defined in this - module. - - If ``tempdir`` is unset or ``None`` at any call to any of the above - functions, Python searches a standard list of directories and sets - *tempdir* to the first one which the calling user can create files in. - The list is: + Python searches a standard list of directories to find one which + the calling user can create files in. The list is: #. The directory named by the :envvar:`TMPDIR` environment variable. @@ -254,12 +225,8 @@ the appropriate function arguments, instead. #. As a last resort, the current working directory. - -.. function:: gettempdir() - - Return the directory currently selected to create temporary files in. If - :data:`tempdir` is not ``None``, this simply returns its contents; otherwise, - the search described above is performed, and the result returned. + The result of this search is cached, see the description of + :data:`tempdir` below. .. function:: gettempdirb() @@ -278,6 +245,23 @@ the appropriate function arguments, instead. .. versionadded:: 3.5 +The module uses a global variable to store the name of the directory +used for temporary files returned by :func:`gettempdir`. It can be +set directly to override the selection process, but this is discouraged. +All functions in this module take a *dir* argument which can be used +to specify the directory and this is the recommend approach. + +.. data:: tempdir + + When set to a value other than ``None``, this variable defines the + default value for the *dir* argument to all the functions defined in this + module. + + If ``tempdir`` is unset or ``None`` at any call to any of the above + functions except :func:`gettempprefix` it is initalized following the + algorithm described in :func:`gettempdir`. + +.. _tempfile-examples: Examples -------- @@ -311,3 +295,42 @@ Here are some examples of typical usage of the :mod:`tempfile` module:: >>> # directory and contents have been removed + +Deprecated functions and variables +---------------------------------- + +A historical way to create temporary files was to first generate a +file name with the :func:`mktemp` function and then create a file +using this name. Unfortunately this is not secure, because a different +process may create a file with this name in the time between the call +to :func:`mktemp` and the subsequent attempt to create the file by the +first process. The solution is to combine the two steps and create the +file immediately. This approach is used by :func:`mkstemp` and the +other functions described above. + +.. function:: mktemp(suffix='', prefix='tmp', dir=None) + + .. deprecated:: 2.3 + Use :func:`mkstemp` instead. + + Return an absolute pathname of a file that did not exist at the time the + call is made. The *prefix*, *suffix*, and *dir* arguments are the same + as for :func:`mkstemp`. + + .. warning:: + + Use of this function may introduce a security hole in your program. By + the time you get around to doing anything with the file name it returns, + someone else may have beaten you to the punch. :func:`mktemp` usage can + be replaced easily with :func:`NamedTemporaryFile`, passing it the + ``delete=False`` parameter:: + + >>> f = NamedTemporaryFile(delete=False) + >>> f.name + '/tmp/tmptjujjt' + >>> f.write(b"Hello World!\n") + 13 + >>> f.close() + >>> os.unlink(f.name) + >>> os.path.exists(f.name) + False diff --git a/Doc/library/threading.rst b/Doc/library/threading.rst index 5269994..c56d707 100644 --- a/Doc/library/threading.rst +++ b/Doc/library/threading.rst @@ -89,7 +89,8 @@ This module defines the following functions: Return the thread stack size used when creating new threads. The optional *size* argument specifies the stack size to be used for subsequently created threads, and must be 0 (use platform or configured default) or a positive - integer value of at least 32,768 (32 KiB). If changing the thread stack size is + integer value of at least 32,768 (32 KiB). If *size* is not specified, + 0 is used. If changing the thread stack size is unsupported, a :exc:`RuntimeError` is raised. If the specified stack size is invalid, a :exc:`ValueError` is raised and the stack size is unmodified. 32 KiB is currently the minimum supported stack size value to guarantee sufficient diff --git a/Doc/library/types.rst b/Doc/library/types.rst index ff75de1..eb27846 100644 --- a/Doc/library/types.rst +++ b/Doc/library/types.rst @@ -86,14 +86,14 @@ Standard names are defined for the following types: .. data:: GeneratorType - The type of :term:`generator`-iterator objects, produced by calling a - generator function. + The type of :term:`generator`-iterator objects, created by + generator functions. .. data:: CoroutineType - The type of :term:`coroutine` objects, produced by calling a - function defined with an :keyword:`async def` statement. + The type of :term:`coroutine` objects, created by + :keyword:`async def` functions. .. versionadded:: 3.5 diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index 8089443..668523b 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -35,7 +35,7 @@ Callable -------- Frameworks expecting callback functions of specific signatures might be -type hinted using `Callable[[Arg1Type, Arg2Type], ReturnType]`. +type hinted using ``Callable[[Arg1Type, Arg2Type], ReturnType]``. For example:: @@ -60,7 +60,7 @@ Since type information about objects kept in containers cannot be statically inferred in a generic way, abstract base classes have been extended to support subscription to denote expected types for container elements. -.. code-block:: python +:: from typing import Mapping, Sequence @@ -70,7 +70,7 @@ subscription to denote expected types for container elements. Generics can be parametrized by using a new factory available in typing called :class:`TypeVar`. -.. code-block:: python +:: from typing import Sequence, TypeVar @@ -85,7 +85,7 @@ User-defined generic types A user-defined class can be defined as a generic class. -.. code-block:: python +:: from typing import TypeVar, Generic from logging import Logger @@ -220,9 +220,7 @@ The module defines the following classes, functions and decorators: Type variables exist primarily for the benefit of static type checkers. They serve as the parameters for generic types as well as for generic function definitions. See class Generic for more - information on generic types. Generic functions work as follows: - - .. code-block:: python + information on generic types. Generic functions work as follows:: def repeat(x: T, n: int) -> Sequence[T]: """Return a list containing n references to x.""" @@ -238,13 +236,13 @@ The module defines the following classes, functions and decorators: the return type is still plain :class:`str`. At runtime, ``isinstance(x, T)`` will raise :exc:`TypeError`. In general, - :func:`isinstance` and :func:`issublass` should not be used with types. + :func:`isinstance` and :func:`issubclass` should not be used with types. Type variables may be marked covariant or contravariant by passing ``covariant=True`` or ``contravariant=True``. See :pep:`484` for more details. By default type variables are invariant. Alternatively, a type variable may specify an upper bound using ``bound=<type>``. - This means that an actual type substituted (explicitly or implictly) + This means that an actual type substituted (explicitly or implicitly) for the type variable must be a subclass of the boundary type, see :pep:`484`. @@ -278,7 +276,7 @@ The module defines the following classes, functions and decorators: * You cannot subclass or instantiate a union. - * You cannot write ``Union[X][Y]`` + * You cannot write ``Union[X][Y]``. * You can use ``Optional[X]`` as a shorthand for ``Union[X, None]``. @@ -331,6 +329,7 @@ The module defines the following classes, functions and decorators: X = TypeVar('X') Y = TypeVar('Y') + def lookup_name(mapping: Mapping[X, Y], key: X, default: Y) -> Y: try: return mapping[key] @@ -347,26 +346,26 @@ The module defines the following classes, functions and decorators: .. class:: SupportsInt - An ABC with one abstract method `__int__`. + An ABC with one abstract method ``__int__``. .. class:: SupportsFloat - An ABC with one abstract method `__float__`. + An ABC with one abstract method ``__float__``. .. class:: SupportsAbs - An ABC with one abstract method `__abs__` that is covariant + An ABC with one abstract method ``__abs__`` that is covariant in its return type. .. class:: SupportsRound - An ABC with one abstract method `__round__` + An ABC with one abstract method ``__round__`` that is covariant in its return type. .. class:: Reversible - An ABC with one abstract method `__reversed__` returning - an `Iterator[T_co]`. + An ABC with one abstract method ``__reversed__`` returning + an ``Iterator[T_co]``. .. class:: Container(Generic[T_co]) @@ -503,9 +502,9 @@ The module defines the following classes, functions and decorators: Return type hints for a function or method object. - This is often the same as obj.__annotations__, but it handles + This is often the same as ``obj.__annotations__``, but it handles forward references encoded as string literals, and if necessary - adds Optional[t] if a default value equal to None is set. + adds ``Optional[t]`` if a default value equal to None is set. .. decorator:: no_type_check(arg) @@ -519,7 +518,7 @@ The module defines the following classes, functions and decorators: .. decorator:: no_type_check_decorator(decorator) - Decorator to give another decorator the @no_type_check effect. + Decorator to give another decorator the :func:`no_type_check` effect. This wraps the decorator with something that wraps the decorated - function in @no_type_check. + function in :func:`no_type_check`. diff --git a/Doc/library/unittest.mock.rst b/Doc/library/unittest.mock.rst index 0f621c8..1b271c8 100644 --- a/Doc/library/unittest.mock.rst +++ b/Doc/library/unittest.mock.rst @@ -552,7 +552,7 @@ the *new_callable* argument to :func:`patch`. keyword arguments (or an empty dictionary). >>> mock = Mock(return_value=None) - >>> print mock.call_args + >>> print(mock.call_args) None >>> mock() >>> mock.call_args @@ -747,7 +747,7 @@ apply to method calls on the mock object. >>> with patch('__main__.Foo.foo', new_callable=PropertyMock) as mock_foo: ... mock_foo.return_value = 'mockity-mock' ... this_foo = Foo() - ... print this_foo.foo + ... print(this_foo.foo) ... this_foo.foo = 6 ... mockity-mock @@ -1135,7 +1135,7 @@ Another use case might be to replace an object with a :class:`io.StringIO` insta >>> from io import StringIO >>> def foo(): - ... print 'Something' + ... print('Something') ... >>> @patch('sys.stdout', new_callable=StringIO) ... def test(mock_stdout): @@ -1249,7 +1249,7 @@ ends. >>> import os >>> with patch.dict('os.environ', {'newkey': 'newvalue'}): - ... print os.environ['newkey'] + ... print(os.environ['newkey']) ... newvalue >>> assert 'newkey' not in os.environ @@ -1462,9 +1462,9 @@ inform the patchers of the different prefix by setting ``patch.TEST_PREFIX``: >>> @patch('__main__.value', 'not three') ... class Thing: ... def foo_one(self): - ... print value + ... print(value) ... def foo_two(self): - ... print value + ... print(value) ... >>> >>> Thing().foo_one() diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst index 5085703..1153f83 100644 --- a/Doc/library/unittest.rst +++ b/Doc/library/unittest.rst @@ -278,8 +278,8 @@ The :option:`-s`, :option:`-p`, and :option:`-t` options can be passed in as positional arguments in that order. The following two command lines are equivalent:: - python -m unittest discover -s project_directory -p '*_test.py' - python -m unittest discover project_directory '*_test.py' + python -m unittest discover -s project_directory -p "*_test.py" + python -m unittest discover project_directory "*_test.py" As well as being a path it is possible to pass a package name, for example ``myproject.subpackage.test``, as the start directory. The package name you diff --git a/Doc/library/xml.etree.elementtree.rst b/Doc/library/xml.etree.elementtree.rst index eef1b58..14e5c99 100644 --- a/Doc/library/xml.etree.elementtree.rst +++ b/Doc/library/xml.etree.elementtree.rst @@ -651,21 +651,29 @@ Element Objects .. attribute:: text + tail - The *text* attribute can be used to hold additional data associated with - the element. As the name implies this attribute is usually a string but - may be any application-specific object. If the element is created from - an XML file the attribute will contain any text found between the element - tags. + These attributes can be used to hold additional data associated with + the element. Their values are usually strings but may be any + application-specific object. If the element is created from + an XML file, the *text* attribute holds either the text between + the element's start tag and its first child or end tag, or ``None``, and + the *tail* attribute holds either the text between the element's + end tag and the next tag, or ``None``. For the XML data + .. code-block:: xml - .. attribute:: tail + <a><b>1<c>2<d/>3</c></b>4</a> - The *tail* attribute can be used to hold additional data associated with - the element. This attribute is usually a string but may be any - application-specific object. If the element is created from an XML file - the attribute will contain any text found after the element's end tag and - before the next tag. + the *a* element has ``None`` for both *text* and *tail* attributes, + the *b* element has *text* ``"1"`` and *tail* ``"4"``, + the *c* element has *text* ``"2"`` and *tail* ``None``, + and the *d* element has *text* ``None`` and *tail* ``"3"``. + + To collect the inner text of an element, see :meth:`itertext`, for + example ``"".join(element.itertext())``. + + Applications may store arbitrary objects in these attributes. .. attribute:: attrib diff --git a/Doc/library/zlib.rst b/Doc/library/zlib.rst index b178fe1..aea0e79 100644 --- a/Doc/library/zlib.rst +++ b/Doc/library/zlib.rst @@ -58,7 +58,7 @@ The available exception and functions in this module are: Raises the :exc:`error` exception if any error occurs. -.. function:: compressobj(level=-1, method=DEFLATED, wbits=15, memlevel=8, strategy=Z_DEFAULT_STRATEGY[, zdict]) +.. function:: compressobj(level=-1, method=DEFLATED, wbits=15, memLevel=8, strategy=Z_DEFAULT_STRATEGY[, zdict]) Returns a compression object, to be used for compressing data streams that won't fit into memory at once. @@ -75,9 +75,9 @@ The available exception and functions in this module are: should be an integer from ``8`` to ``15``. Higher values give better compression, but use more memory. - *memlevel* controls the amount of memory used for internal compression state. - Valid values range from ``1`` to ``9``. Higher values using more memory, - but are faster and produce smaller output. + The *memLevel* argument controls the amount of memory used for the + internal compression state. Valid values range from ``1`` to ``9``. + Higher values use more memory, but are faster and produce smaller output. *strategy* is used to tune the compression algorithm. Possible values are ``Z_DEFAULT_STRATEGY``, ``Z_FILTERED``, and ``Z_HUFFMAN_ONLY``. diff --git a/Doc/tools/susp-ignored.csv b/Doc/tools/susp-ignored.csv index 7664f92..9e40fc6 100644 --- a/Doc/tools/susp-ignored.csv +++ b/Doc/tools/susp-ignored.csv @@ -10,7 +10,6 @@ extending/embedding,,:numargs,"if(!PyArg_ParseTuple(args, "":numargs""))" extending/extending,,:myfunction,"PyArg_ParseTuple(args, ""D:myfunction"", &c);" extending/extending,,:set,"if (PyArg_ParseTuple(args, ""O:set_callback"", &temp)) {" extending/newtypes,,:call,"if (!PyArg_ParseTuple(args, ""sss:call"", &arg1, &arg2, &arg3)) {" -extending/windows,,:initspam,/export:initspam faq/programming,,:chr,">=4.0) or 1+f(xc,yc,x*x-y*y+xc,2.0*x*y+yc,k-1,f):f(xc,yc,x,y,k,f):chr(" faq/programming,,::,for x in sequence[::-1]: faq/programming,,:reduce,"print((lambda Ru,Ro,Iu,Io,IM,Sx,Sy:reduce(lambda x,y:x+y,map(lambda y," @@ -288,6 +287,6 @@ library/xml.etree.elementtree,332,:character,"for char in actor.findall('role:ch library/zipapp,31,:main,"$ python -m zipapp myapp -m ""myapp:main""" library/zipapp,82,:fn,"argument should have the form ""pkg.mod:fn"", where ""pkg.mod"" is a" library/zipapp,155,:callable,"""pkg.module:callable"" and the archive will be run by importing" -library/stdtypes,3767,::,>>> m[::2].tolist() +library/stdtypes,,::,>>> m[::2].tolist() library/sys,1115,`,# ``wrapper`` creates a ``wrap(coro)`` coroutine: tutorial/venv,77,:c7b9645a6f35,"Python 3.4.3+ (3.4:c7b9645a6f35+, May 22 2015, 09:31:25)" diff --git a/Doc/tutorial/datastructures.rst b/Doc/tutorial/datastructures.rst index a2031ed..0d51480 100644 --- a/Doc/tutorial/datastructures.rst +++ b/Doc/tutorial/datastructures.rst @@ -612,18 +612,18 @@ returns a new sorted list while leaving the source unaltered. :: orange pear -To change a sequence you are iterating over while inside the loop (for -example to duplicate certain items), it is recommended that you first make -a copy. Looping over a sequence does not implicitly make a copy. The slice -notation makes this especially convenient:: - - >>> words = ['cat', 'window', 'defenestrate'] - >>> for w in words[:]: # Loop over a slice copy of the entire list. - ... if len(w) > 6: - ... words.insert(0, w) +It is sometimes tempting to change a list while you are looping over it; +however, it is often simpler and safer to create a new list instead. :: + + >>> import math + >>> raw_data = [56.2, float('NaN'), 51.7, 55.3, 52.5, float('NaN'), 47.8] + >>> filtered_data = [] + >>> for value in raw_data: + ... if not math.isnan(value): + ... filtered_data.append(value) ... - >>> words - ['defenestrate', 'cat', 'window', 'defenestrate'] + >>> filtered_data + [56.2, 51.7, 55.3, 52.5, 47.8] .. _tut-conditions: diff --git a/Doc/using/mac.rst b/Doc/using/mac.rst index ede864d..ef091e5 100644 --- a/Doc/using/mac.rst +++ b/Doc/using/mac.rst @@ -102,8 +102,9 @@ Configuration Python on OS X honors all standard Unix environment variables such as :envvar:`PYTHONPATH`, but setting these variables for programs started from the Finder is non-standard as the Finder does not read your :file:`.profile` or -:file:`.cshrc` at startup. You need to create a file :file:`~ -/.MacOSX/environment.plist`. See Apple's Technical Document QA1067 for details. +:file:`.cshrc` at startup. You need to create a file +:file:`~/.MacOSX/environment.plist`. See Apple's Technical Document QA1067 for +details. For more information on installation Python packages in MacPython, see section :ref:`mac-package-manager`. diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst index 5c19b81..ffaaa79 100644 --- a/Doc/using/windows.rst +++ b/Doc/using/windows.rst @@ -77,8 +77,8 @@ machines without user interaction. These options may also be set without suppressing the UI in order to change some of the defaults. To completely hide the installer UI and install Python silently, pass the -``/quiet`` (``/q``) option. To skip past the user interaction but still display -progress and errors, pass the ``/passive`` (``/p``) option. The ``/uninstall`` +``/quiet`` option. To skip past the user interaction but still display +progress and errors, pass the ``/passive`` option. The ``/uninstall`` option may be passed to immediately begin removing Python - no prompt will be displayed. diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst index 11f871b..2acf024 100644 --- a/Doc/whatsnew/3.5.rst +++ b/Doc/whatsnew/3.5.rst @@ -4,6 +4,7 @@ :Release: |release| :Date: |today| +:Editors: Elvis Pranskevichus <elprans@gmail.com>, Yury Selivanov <yselivanov@gmail.com> .. Rules for maintenance: @@ -46,7 +47,6 @@ when researching a change. This article explains the new features in Python 3.5, compared to 3.4. - For full details, see the :source:`Misc/NEWS` file. .. note:: @@ -69,8 +69,8 @@ Summary -- Release highlights New syntax features: -* :pep:`465`, a new matrix multiplication operator: ``a @ b``. * :pep:`492`, coroutines with async and await syntax. +* :pep:`465`, a new matrix multiplication operator: ``a @ b``. * :pep:`448`, additional unpacking generalizations. New library modules: @@ -81,175 +81,308 @@ New library modules: New built-in features: * ``bytes % args``, ``bytearray % args``: :pep:`461` - Adding ``%`` formatting - to bytes and bytearray + to bytes and bytearray. + * ``b'\xf0\x9f\x90\x8d'.hex()``, ``bytearray(b'\xf0\x9f\x90\x8d').hex()``, ``memoryview(b'\xf0\x9f\x90\x8d').hex()``: :issue:`9951` - A ``hex`` method has been added to bytes, bytearray, and memoryview. + +* :class:`memoryview` (including multi-dimensional) now supports tuple indexing. + (Contributed by Antoine Pitrou in :issue:`23632`.) + * Generators have new ``gi_yieldfrom`` attribute, which returns the object being iterated by ``yield from`` expressions. (Contributed by Benno Leslie and Yury Selivanov in :issue:`24450`.) -* New :exc:`RecursionError` exception. (Contributed by Georg Brandl + +* New :exc:`RecursionError` exception. (Contributed by Georg Brandl in :issue:`19235`.) -Implementation improvements: +* New :exc:`StopAsyncIteration` exception. (Contributed by + Yury Selivanov in :issue:`24017`. See also :pep:`492`.) + +CPython implementation improvements: * When the ``LC_TYPE`` locale is the POSIX locale (``C`` locale), :py:data:`sys.stdin` and :py:data:`sys.stdout` are now using the - ``surrogateescape`` error handler, instead of the ``strict`` error handler - (:issue:`19977`). + ``surrogateescape`` error handler, instead of the ``strict`` error handler. + (Contributed by Victor Stinner in :issue:`19977`.) + +* ``.pyo`` files are no longer used and have been replaced by a more flexible + scheme that inclides the optimization level explicitly in ``.pyc`` name. + (:pep:`488`) -* :pep:`488`, the elimination of ``.pyo`` files. -* :pep:`489`, multi-phase initialization of extension modules. +* Builtin and extension modules are now initialized in a multi-phase process, + which is similar to how Python modules are loaded. (:pep:`489`). Significantly Improved Library Modules: -* :class:`collections.OrderedDict` is now implemented in C, which improves - its performance between 4x to 100x times. Contributed by Eric Snow in - :issue:`16991`. +* :class:`collections.OrderedDict` is now implemented in C, which makes it + 4 to 100 times faster. (Contributed by Eric Snow in :issue:`16991`.) * You may now pass bytes to the :mod:`tempfile` module's APIs and it will - return the temporary pathname as bytes instead of str. It also accepts - a value of ``None`` on parameters where only str was accepted in the past to - do the right thing based on the types of the other inputs. Two functions, - :func:`gettempdirb` and :func:`gettempprefixb`, have been added to go along - with this. This behavior matches that of the :mod:`os` APIs. + return the temporary pathname as :class:`bytes` instead of :class:`str`. + It also accepts a value of ``None`` on parameters where only str was + accepted in the past to do the right thing based on the types of the + other inputs. Two functions, :func:`gettempdirb` and + :func:`gettempprefixb`, have been added to go along with this. + This behavior matches that of the :mod:`os` APIs. + (Contributed by Gregory P. Smith in :issue:`24230`.) * :mod:`ssl` module gained support for Memory BIO, which decouples SSL protocol handling from network IO. (Contributed by Geert Jansen in :issue:`21965`.) +* :mod:`traceback` has new lightweight and convenient to work with + classes :class:`~traceback.TracebackException`, + :class:`~traceback.StackSummary`, and :class:`~traceback.FrameSummary`. + (Contributed by Robert Collins in :issue:`17911`.) + +* Most of :func:`functools.lru_cache` machinery is now implemented in C. + (Contributed by Matt Joiner, Alexey Kachayev, and Serhiy Storchaka + in :issue:`14373`.) + Security improvements: -* None yet. +* SSLv3 is now disabled throughout the standard library. + It can still be enabled by instantiating a :class:`ssl.SSLContext` + manually. (See :issue:`22638` for more details; this change was + backported to CPython 3.4 and 2.7.) + +* HTTP cookie parsing is now stricter, in order to protect + against potential injection attacks. (Contributed by Antoine Pitrou + in :issue:`22796`.) Windows improvements: -* A new installer for Windows has replaced the old MSI. See :ref:`using-on-windows` - for more information. +* A new installer for Windows has replaced the old MSI. + See :ref:`using-on-windows` for more information. + * Windows builds now use Microsoft Visual C++ 14.0, and extension modules should use the same. +Please read on for a comprehensive list of user-facing changes, including many +other smaller improvements, CPython optimizations, deprecations, and potential +porting issues. -Please read on for a comprehensive list of user-facing changes. +New Features +============ -.. PEP-sized items next. +.. _whatsnew-pep-492: -.. _pep-4XX: +PEP 492 - Coroutines with async and await syntax +------------------------------------------------ -.. PEP 4XX: Virtual Environments -.. ============================= +:pep:`492` greatly improves support for asynchronous programming in Python +by adding :term:`awaitable objects <awaitable>`, +:term:`coroutine functions <coroutine function>`, +:term:`asynchronous iteration <asynchronous iterable>`, +and :term:`asynchronous context managers <asynchronous context manager>`. +Coroutine functions are declared using the new :keyword:`async def` syntax:: -.. (Implemented by Foo Bar.) + >>> async def coro(): + ... return 'spam' -.. .. seealso:: +Inside a coroutine function, the new :keyword:`await` expression can be used +to suspend coroutine execution until the result is available. Any object +can be *awaited*, as long as it implements the :term:`awaitable` protocol by +defining the :meth:`__await__` method. - :pep:`4XX` - Python Virtual Environments - PEP written by Carl Meyer +PEP 492 also adds :keyword:`async for` statement for convenient iteration +over asynchronous iterables. +An example of a simple HTTP client written using the new syntax:: -PEP 492 - Coroutines with async and await syntax ------------------------------------------------- + import asyncio -The PEP added dedicated syntax for declaring :term:`coroutines <coroutine>`, -:keyword:`await` expressions, new asynchronous :keyword:`async for` -and :keyword:`async with` statements. + async def http_get(domain): + reader, writer = await asyncio.open_connection(domain, 80) -Example:: + writer.write(b'\r\n'.join([ + b'GET / HTTP/1.1', + b'Host: %b' % domain.encode('latin-1'), + b'Connection: close', + b'', b'' + ])) - async def read_data(db): - async with db.transaction(): - data = await db.fetch('SELECT ...') + async for line in reader: + print('>>>', line) -PEP written and implemented by Yury Selivanov. + writer.close() -.. seealso:: + loop = asyncio.get_event_loop() + try: + loop.run_until_complete(http_get('example.com')) + finally: + loop.close() - :pep:`492` -- Coroutines with async and await syntax +Similarly to asynchronous iteration, there is a new syntax for asynchronous +context managers. The following script:: -PEP 461 - Formatting support for bytes and bytearray ----------------------------------------------------- + import asyncio -This PEP proposes adding % formatting operations similar to Python 2's ``str`` -type to :class:`bytes` and :class:`bytearray`. + async def coro(name, lock): + print('coro {}: waiting for lock'.format(name)) + async with lock: + print('coro {}: holding the lock'.format(name)) + await asyncio.sleep(1) + print('coro {}: releasing the lock'.format(name)) -Examples:: + loop = asyncio.get_event_loop() + lock = asyncio.Lock() + coros = asyncio.gather(coro(1, lock), coro(2, lock)) + try: + loop.run_until_complete(coros) + finally: + loop.close() - >>> b'Hello %s!' % b'World' - b'Hello World!' - >>> b'x=%i y=%f' % (1, 2.5) - b'x=1 y=2.500000' +will print:: -Unicode is not allowed for ``%s``, but it is accepted by ``%a`` (equivalent of -``repr(obj).encode('ascii', 'backslashreplace')``):: + coro 2: waiting for lock + coro 2: holding the lock + coro 1: waiting for lock + coro 2: releasing the lock + coro 1: holding the lock + coro 1: releasing the lock - >>> b'Hello %s!' % 'World' - Traceback (most recent call last): - File "<stdin>", line 1, in <module> - TypeError: %b requires bytes, or an object that implements __bytes__, not 'str' - >>> b'price: %a' % '10€' - b"price: '10\\u20ac'" +Note that both :keyword:`async for` and :keyword:`async with` can only +be used inside a coroutine function declared with :keyword:`async def`. + +Coroutine functions are intended to be run inside a compatible event loop, +such as :class:`asyncio.Loop`. .. seealso:: - :pep:`461` -- Adding % formatting to bytes and bytearray + :pep:`492` -- Coroutines with async and await syntax + PEP written and implemented by Yury Selivanov. +.. _whatsnew-pep-465: + PEP 465 - A dedicated infix operator for matrix multiplication -------------------------------------------------------------- -This PEP proposes a new binary operator to be used for matrix multiplication, -called ``@``. (Mnemonic: ``@`` is ``*`` for mATrices.) +:pep:`465` adds the ``@`` infix operator for matrix multiplication. +Currently, no builtin Python types implement the new operator, however, it +can be implemented by defining :meth:`__matmul__`, :meth:`__rmatmul__`, +and :meth:`__imatmul__` for regular, reflected, and in-place matrix +multiplication. The semantics of these methods is similar to that of +methods defining other infix arithmetic operators. + +Matrix multiplication is a notably common operation in many fields of +mathematics, science, engineering, and the addition of ``@`` allows writing +cleaner code:: + + S = (H @ beta - r).T @ inv(H @ V @ H.T) @ (H @ beta - r) + +instead of:: + + S = dot((dot(H, beta) - r).T, + dot(inv(dot(dot(H, V), H.T)), dot(H, beta) - r)) + +An upcoming release of NumPy 1.10 will add support for the new operator:: + + >>> import numpy + + >>> x = numpy.ones(3) + >>> x + array([ 1., 1., 1.]) + + >>> m = numpy.eye(3) + >>> m + array([[ 1., 0., 0.], + [ 0., 1., 0.], + [ 0., 0., 1.]]) + + >>> x @ m + array([ 1., 1., 1.]) + .. seealso:: :pep:`465` -- A dedicated infix operator for matrix multiplication + PEP written by Nathaniel J. Smith; implemented by Benjamin Peterson. +.. _whatsnew-pep-448: + PEP 448 - Additional Unpacking Generalizations ---------------------------------------------- -This PEP proposes extended usages of the ``*`` iterable unpacking -operator and ``**`` dictionary unpacking operators -to allow unpacking in more positions, an arbitrary number of -times, and in additional circumstances. Specifically, -in function calls, in comprehensions and generator expressions, and -in displays. +:pep:`448` extends the allowed uses of the ``*`` iterable unpacking +operator and ``**`` dictionary unpacking operator. It is now possible +to use an arbitrary number of unpackings in function calls:: + + >>> print(*[1], *[2], 3, *[4, 5]) + 1 2 3 4 5 -Function calls are proposed to support an arbitrary number of -unpackings rather than just one:: + >>> def fn(a, b, c, d): + ... print(a, b, c, d) + ... - >>> print(*[1], *[2], 3) - 1 2 3 - >>> dict(**{'x': 1}, y=2, **{'z': 3}) - {'x': 1, 'y': 2, 'z': 3} + >>> fn(**{'a': 1, 'c': 3}, **{'b': 2, 'd': 4}) + 1 2 3 4 -Unpacking is proposed to be allowed inside tuple, list, set, -and dictionary displays:: +Similarly, tuple, list, set, and dictionary displays allow multiple +unpackings:: >>> *range(4), 4 (0, 1, 2, 3, 4) + >>> [*range(4), 4] [0, 1, 2, 3, 4] - >>> {*range(4), 4} - {0, 1, 2, 3, 4} + + >>> {*range(4), 4, *(5, 6, 7)} + {0, 1, 2, 3, 4, 5, 6, 7} + >>> {'x': 1, **{'y': 2}} {'x': 1, 'y': 2} -In dictionaries, later values will always override earlier ones:: +.. seealso:: + + :pep:`448` -- Additional Unpacking Generalizations + PEP written by Joshua Landau; implemented by Neil Girdhar, + Thomas Wouters, and Joshua Landau. + + +.. _whatsnew-pep-461: - >>> {'x': 1, **{'x': 2}} - {'x': 2} +PEP 461 - % formatting support for bytes and bytearray +------------------------------------------------------ - >>> {**{'x': 2}, 'x': 1} - {'x': 1} +PEP 461 adds % formatting to :class:`bytes` and :class:`bytearray`, aiding in +handling data that is a mixture of binary and ASCII compatible text. This +feature also eases porting such code from Python 2. + +Examples:: + + >>> b'Hello %s!' % b'World' + b'Hello World!' + + >>> b'x=%i y=%f' % (1, 2.5) + b'x=1 y=2.500000' + +Unicode is not allowed for ``%s``, but it is accepted by ``%a`` (equivalent of +``repr(obj).encode('ascii', 'backslashreplace')``):: + + >>> b'Hello %s!' % 'World' + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + TypeError: %b requires bytes, or an object that implements __bytes__, not 'str' + + >>> b'price: %a' % '10€' + b"price: '10\\u20ac'" .. seealso:: - :pep:`448` -- Additional Unpacking Generalizations + :pep:`461` -- Adding % formatting to bytes and bytearray + PEP written by Ethan Furman; implemented by Neil Schemenauer and + Ethan Furman. + +.. _whatsnew-pep-484: PEP 484 - Type Hints -------------------- @@ -261,8 +394,8 @@ where annotations are not available. For example, here is a simple function whose argument and return type are declared in the annotations:: - def greeting(name: str) -> str: - return 'Hello ' + name + def greeting(name: str) -> str: + return 'Hello ' + name The type system supports unions, generic types, and a special type named ``Any`` which is consistent with (i.e. assignable to and from) all @@ -270,25 +403,29 @@ types. .. seealso:: - * :pep:`484` -- Type Hints * :mod:`typing` module documentation + * :pep:`484` -- Type Hints + PEP written by Guido van Rossum, Jukka Lehtosalo, and Łukasz Langa; + implemented by Guido van Rossum. + +.. _whatsnew-pep-471: PEP 471 - os.scandir() function -- a better and faster directory iterator ------------------------------------------------------------------------- :pep:`471` adds a new directory iteration function, :func:`os.scandir`, -to the standard library. Additionally, :func:`os.walk` is now +to the standard library. Additionally, :func:`os.walk` is now implemented using :func:`os.scandir`, which speeds it up by 3-5 times on POSIX systems and by 7-20 times on Windows systems. -PEP and implementation written by Ben Hoyt with the help of Victor Stinner. - .. seealso:: - :pep:`471` -- os.scandir() function -- a better and faster directory - iterator + :pep:`471` -- os.scandir() function -- a better and faster directory iterator + PEP written and implemented by Ben Hoyt with the help of Victor Stinner. + +.. _whatsnew-pep-475: PEP 475: Retry system calls failing with EINTR ---------------------------------------------- @@ -302,69 +439,48 @@ Examples of functions which are now retried when interrupted by a signal instead of raising :exc:`InterruptedError` if the Python signal handler does not raise an exception: -* :func:`open`, :func:`os.open`, :func:`io.open` -* functions of the :mod:`faulthandler` module -* :mod:`os` functions: - - - :func:`os.fchdir` - - :func:`os.fchmod` - - :func:`os.fchown` - - :func:`os.fdatasync` - - :func:`os.fstat` - - :func:`os.fstatvfs` - - :func:`os.fsync` - - :func:`os.ftruncate` - - :func:`os.mkfifo` - - :func:`os.mknod` - - :func:`os.posix_fadvise` - - :func:`os.posix_fallocate` - - :func:`os.pread` - - :func:`os.pwrite` - - :func:`os.read` - - :func:`os.readv` - - :func:`os.sendfile` - - :func:`os.wait3` - - :func:`os.wait4` - - :func:`os.wait` - - :func:`os.waitid` - - :func:`os.waitpid` - - :func:`os.write` - - :func:`os.writev` - - special cases: :func:`os.close` and :func:`os.dup2` now ignore - :py:data:`~errno.EINTR` error, the syscall is not retried (see the PEP - for the rationale) - -* :mod:`select` functions: - - - :func:`select.devpoll.poll` - - :func:`select.epoll.poll` - - :func:`select.kqueue.control` - - :func:`select.poll.poll` - - :func:`select.select` - -* :func:`socket.socket` methods: - - - :meth:`~socket.socket.accept` - - :meth:`~socket.socket.connect` (except for non-blocking sockets) - - :meth:`~socket.socket.recv` - - :meth:`~socket.socket.recvfrom` - - :meth:`~socket.socket.recvmsg` - - :meth:`~socket.socket.send` - - :meth:`~socket.socket.sendall` - - :meth:`~socket.socket.sendmsg` - - :meth:`~socket.socket.sendto` - -* :func:`signal.sigtimedwait`, :func:`signal.sigwaitinfo` -* :func:`time.sleep` - -PEP and implementation written by Charles-François Natali and Victor Stinner, -with the help of Antoine Pitrou (the french connection). +* :func:`open`, :func:`os.open`, :func:`io.open`; + +* functions of the :mod:`faulthandler` module; + +* :mod:`os` functions: :func:`~os.fchdir`, :func:`~os.fchmod`, + :func:`~os.fchown`, :func:`~os.fdatasync`, :func:`~os.fstat`, + :func:`~os.fstatvfs`, :func:`~os.fsync`, :func:`~os.ftruncate`, + :func:`~os.mkfifo`, :func:`~os.mknod`, :func:`~os.posix_fadvise`, + :func:`~os.posix_fallocate`, :func:`~os.pread`, :func:`~os.pwrite`, + :func:`~os.read`, :func:`~os.readv`, :func:`~os.sendfile`, + :func:`~os.wait3`, :func:`~os.wait4`, :func:`~os.wait`, + :func:`~os.waitid`, :func:`~os.waitpid`, :func:`~os.write`, + :func:`~os.writev`; + +* special cases: :func:`os.close` and :func:`os.dup2` now ignore + :py:data:`~errno.EINTR` error, the syscall is not retried (see the PEP + for the rationale); + +* :mod:`select` functions: :func:`~select.devpoll.poll`, + :func:`~select.epoll.poll`, :func:`~select.kqueue.control`, + :func:`~select.poll.poll`, :func:`~select.select`; + +* :func:`socket.socket` methods: :meth:`~socket.socket.accept`, + :meth:`~socket.socket.connect` (except for non-blocking sockets), + :meth:`~socket.socket.recv`, :meth:`~socket.socket.recvfrom`, + :meth:`~socket.socket.recvmsg`, :meth:`~socket.socket.send`, + :meth:`~socket.socket.sendall`, :meth:`~socket.socket.sendmsg`, + :meth:`~socket.socket.sendto`; + +* :func:`signal.sigtimedwait`, :func:`signal.sigwaitinfo`; + +* :func:`time.sleep`. .. seealso:: :pep:`475` -- Retry system calls failing with EINTR + PEP and implementation written by Charles-François Natali and + Victor Stinner, with the help of Antoine Pitrou (the french connection). +.. _whatsnew-pep-479: + PEP 479: Change StopIteration handling inside generators -------------------------------------------------------- @@ -378,14 +494,15 @@ be used:: Without a ``__future__`` import, a :exc:`PendingDeprecationWarning` will be raised. -PEP written by Chris Angelico and Guido van Rossum. Implemented by -Chris Angelico, Yury Selivanov and Nick Coghlan. - .. seealso:: :pep:`479` -- Change StopIteration handling inside generators + PEP written by Chris Angelico and Guido van Rossum. Implemented by + Chris Angelico, Yury Selivanov and Nick Coghlan. +.. _whatsnew-pep-486: + PEP 486: Make the Python Launcher aware of virtual environments --------------------------------------------------------------- @@ -397,7 +514,10 @@ environment will be used. .. seealso:: :pep:`486` -- Make the Python Launcher aware of virtual environments + PEP written and implemented by Paul Moore. + +.. _whatsnew-pep-488: PEP 488: Elimination of PYO files --------------------------------- @@ -407,14 +527,18 @@ PEP 488: Elimination of PYO files need to constantly regenerate bytecode files, ``.pyc`` files now have an optional ``opt-`` tag in their name when the bytecode is optimized. This has the side-effect of no more bytecode file name clashes when running under either -``-O`` or ``-OO``. Consequently, bytecode files generated from ``-O``, and -``-OO`` may now exist simultaneously. :func:`importlib.util.cache_from_source` -has an updated API to help with this change. +:option:`-O` or :option:`-OO`. Consequently, bytecode files generated from +:option:`-O`, and :option:`-OO` may now exist simultaneously. +:func:`importlib.util.cache_from_source` has an updated API to help with +this change. .. seealso:: :pep:`488` -- Elimination of PYO files + PEP written and implemented by Brett Cannon. + +.. _whatsnew-pep-489: PEP 489: Multi-phase extension module initialization ---------------------------------------------------- @@ -429,7 +553,12 @@ rather than being restricted to ASCII. .. seealso:: - :pep:`488` -- Multi-phase extension module initialization + :pep:`489` -- Multi-phase extension module initialization + PEP written by Petr Viktorin, Stefan Behnel, and Nick Coghlan; + implemented by Petr Viktorin. + + +.. _whatsnew-pep-485: PEP 485: A function for testing approximate equality ---------------------------------------------------- @@ -442,18 +571,21 @@ close is determined according to given absolute and relative tolerances. .. seealso:: :pep:`485` -- A function for testing approximate equality + PEP written by Christopher Barker; implemented by Chris Barker and + Tal Einat. + Other Language Changes ====================== Some smaller changes made to the core Python language are: -* Added the ``'namereplace'`` error handlers. The ``'backslashreplace'`` +* Added the ``"namereplace"`` error handlers. The ``"backslashreplace"`` error handlers now works with decoding and translating. (Contributed by Serhiy Storchaka in :issue:`19676` and :issue:`22286`.) * The :option:`-b` option now affects comparisons of :class:`bytes` with - :class:`int`. (Contributed by Serhiy Storchaka in :issue:`23681`) + :class:`int`. (Contributed by Serhiy Storchaka in :issue:`23681`.) * New Kazakh :ref:`codec <standard-encodings>` ``kz1048``. (Contributed by Serhiy Storchaka in :issue:`22682`.) @@ -465,6 +597,9 @@ Some smaller changes made to the core Python language are: * New Tajik :ref:`codec <standard-encodings>` ``koi8_t``. (Contributed by Serhiy Storchaka in :issue:`22681`.) +* Circular imports involving relative imports are now supported. + (Contributed by Brett Cannon and Antoine Pitrou in :issue:`17636`.) + New Modules =========== @@ -476,8 +611,8 @@ zipapp The new :mod:`zipapp` module (specified in :pep:`441`) provides an API and command line tool for creating executable Python Zip Applications, which -were introduced in Python 2.6 in :issue:`1739468` but which were not well -publicised, either at the time or since. +were introduced in Python 2.6 in :issue:`1739468`, but which were not well +publicized, either at the time or since. With the new module, bundling your application is as simple as putting all the files, including a ``__main__.py`` file, into a directory ``myapp`` @@ -486,6 +621,13 @@ and running:: $ python -m zipapp myapp $ python myapp.pyz +The module implementation has been contributed by Paul Moore in +:issue:`23491`. + +.. seealso:: + + :pep:`441` -- Improving Python ZIP Application Support + Improved Modules ================ @@ -493,529 +635,1135 @@ Improved Modules argparse -------- -* :class:`~argparse.ArgumentParser` now allows to disable - :ref:`abbreviated usage <prefix-matching>` of long options by setting - :ref:`allow_abbrev` to ``False``. - (Contributed by Jonathan Paugh, Steven Bethard, paul j3 and Daniel Eriksson.) +The :class:`~argparse.ArgumentParser` class now allows to disable +:ref:`abbreviated usage <prefix-matching>` of long options by setting +:ref:`allow_abbrev` to ``False``. (Contributed by Jonathan Paugh, +Steven Bethard, paul j3 and Daniel Eriksson in :issue:`14910`.) + + +bz2 +--- + +The :meth:`BZ2Decompressor.decompress <bz2.BZ2Decompressor.decompress>` +method now accepts an optional *max_length* argument to limit the maximum +size of decompressed data. (Contributed by Nikolaus Rath in :issue:`15955`.) + cgi --- -* :class:`~cgi.FieldStorage` now supports the context management protocol. - (Contributed by Berker Peksag in :issue:`20289`.) +The :class:`~cgi.FieldStorage` class now supports the context management +protocol. (Contributed by Berker Peksag in :issue:`20289`.) + cmath ----- -* :func:`cmath.isclose` function added. - (Contributed by Chris Barker and Tal Einat in :issue:`24270`.) +A new function :func:`~cmath.isclose` provides a way to test for approximate +equality. (Contributed by Chris Barker and Tal Einat in :issue:`24270`.) code ---- -* The :func:`code.InteractiveInterpreter.showtraceback` method now prints - the full chained traceback, just like the interactive interpreter. - (Contributed by Claudiu Popa in :issue:`17442`.) +The :func:`InteractiveInterpreter.showtraceback <code.InteractiveInterpreter.showtraceback>` +method now prints the full chained traceback, just like the interactive +interpreter. (Contributed by Claudiu Popa in :issue:`17442`.) + collections ----------- -* You can now update docstrings produced by :func:`collections.namedtuple`:: +The :class:`~collections.OrderedDict` class is now implemented in C, which +makes it 4 to 100 times faster. (Contributed by Eric Snow in :issue:`16991`.) + +:meth:`OrderedDict.items <collections.OrderedDict.items>`, +:meth:`OrderedDict.keys <collections.OrderedDict.keys>`, +:meth:`OrderedDict.values <collections.OrderedDict.values>` views now support +:func:`reversed` iteration. +(Contributed by Serhiy Storchaka in :issue:`19505`.) + +The :class:`~collections.deque` class now defines +:meth:`~collections.deque.index`, :meth:`~collections.deque.insert`, and +:meth:`~collections.deque.copy`, as well as supports ``+`` and ``*`` operators. +This allows deques to be recognized as a :class:`~collections.abc.MutableSequence` +and improves their substitutability for lists. +(Contributed by Raymond Hettinger :issue:`23704`.) + +Docstrings produced by :func:`~collections.namedtuple` can now be updated:: Point = namedtuple('Point', ['x', 'y']) Point.__doc__ = 'ordered pair' Point.x.__doc__ = 'abscissa' Point.y.__doc__ = 'ordinate' - (Contributed by Berker Peksag in :issue:`24064`.) +(Contributed by Berker Peksag in :issue:`24064`.) + +The :class:`~collections.UserString` class now implements +:meth:`__getnewargs__`, :meth:`__rmod__`, :meth:`~str.casefold`, +:meth:`~str.format_map`, :meth:`~str.isprintable`, and :meth:`~str.maketrans` +methods to match corresponding methods of :class:`str`. +(Contributed by Joe Jevnik in :issue:`22189`.) + + +collections.abc +--------------- + +A new :class:`~collections.abc.Generator` abstract base class. (Contributed +by Stefan Behnel in :issue:`24018`.) + +New :class:`~collections.abc.Coroutine`, +:class:`~collections.abc.AsyncIterator`, and +:class:`~collections.abc.AsyncIterable` abstract base classes. +(Contributed by Yury Selivanov in :issue:`24184`.) + compileall ---------- -* :func:`compileall.compile_dir` and :mod:`compileall`'s command-line interface - can now do parallel bytecode compilation. - (Contributed by Claudiu Popa in :issue:`16104`.) +A new :mod:`compileall` option, ``-j N``, allows to run ``N`` workers +sumultaneously to perform parallel bytecode compilation. +The :func:`~compileall.compile_dir` function has a corresponding ``workers`` +parameter. (Contributed by Claudiu Popa in :issue:`16104`.) + +The ``-q`` command line option can now be specified more than once, in +which case all output, including errors, will be suppressed. The corresponding +``quiet`` parameter in :func:`~compileall.compile_dir`, +:func:`~compileall.compile_file`, and :func:`~compileall.compile_path` can now +accept an integer value indicating the level of output suppression. +(Contributed by Thomas Kluyver in :issue:`21338`.) + + +concurrent.futures +------------------ + +The :meth:`Executor.map <concurrent.futures.Executor.map>` method now accepts a +*chunksize* argument to allow batching of tasks to improve performance when +:meth:`~concurrent.futures.ProcessPoolExecutor` is used. +(Contributed by Dan O'Reilly in :issue:`11271`.) + contextlib ---------- -* The new :func:`contextlib.redirect_stderr` context manager(similar to - :func:`contextlib.redirect_stdout`) makes it easier for utility scripts to - handle inflexible APIs that write their output to :data:`sys.stderr` and - don't provide any options to redirect it. - (Contributed by Berker Peksag in :issue:`22389`.) +The new :func:`~contextlib.redirect_stderr` context manager (similar to +:func:`~contextlib.redirect_stdout`) makes it easier for utility scripts to +handle inflexible APIs that write their output to :data:`sys.stderr` and +don't provide any options to redirect it. (Contributed by Berker Peksag in +:issue:`22389`.) + curses ------ -* The new :func:`curses.update_lines_cols` function updates the variables - :envvar:`curses.LINES` and :envvar:`curses.COLS`. + +The new :func:`~curses.update_lines_cols` function updates :envvar:`LINES` +and :envvar:`COLS` environment variables. This is useful for detecting +manual screen resize. (Contributed by Arnon Yaari in :issue:`4254`.) + difflib ------- -* The charset of the HTML document generated by :meth:`difflib.HtmlDiff.make_file` - can now be customized by using *charset* keyword-only parameter. The default - charset of HTML document changed from ``'ISO-8859-1'`` to ``'utf-8'``. - (Contributed by Berker Peksag in :issue:`2052`.) +The charset of HTML documents generated by +:meth:`HtmlDiff.make_file <difflib.HtmlDiff.make_file>` +can now be customized by using a new *charset* keyword-only argument. +The default charset of HTML document changed from ``"ISO-8859-1"`` +to ``"utf-8"``. +(Contributed by Berker Peksag in :issue:`2052`.) + +The :func:`~difflib.diff_bytes` function can now compare lists of byte +strings. This fixes a regression from Python 2. +(Contributed by Terry J. Reedy and Greg Ward in :issue:`17445`.) -* It's now possible to compare lists of byte strings with - :func:`difflib.diff_bytes` (fixes a regression from Python 2). distutils --------- -* The ``build`` and ``build_ext`` commands now accept a ``-j`` - option to enable parallel building of extension modules. - (Contributed by Antoine Pitrou in :issue:`5309`.) +Both ``build`` and ``build_ext`` commands now accept a ``-j`` option to +enable parallel building of extension modules. +(Contributed by Antoine Pitrou in :issue:`5309`.) + +The :mod:`distutils` module now supports ``xz`` compression, and can be +enabled by passing ``xztar`` as an argument to ``bdist --format``. +(Contributed by Serhiy Storchaka in :issue:`16314`.) -* Added support for the LZMA compression. - (Contributed by Serhiy Storchaka in :issue:`16314`.) doctest ------- -* :func:`doctest.DocTestSuite` returns an empty :class:`unittest.TestSuite` if - *module* contains no docstrings instead of raising :exc:`ValueError`. - (Contributed by Glenn Jones in :issue:`15916`.) +The :func:`~doctest.DocTestSuite` function returns an empty +:class:`unittest.TestSuite` if *module* contains no docstrings instead of +raising :exc:`ValueError`. (Contributed by Glenn Jones in :issue:`15916`.) + email ----- -* A new policy option :attr:`~email.policy.Policy.mangle_from_` controls - whether or not lines that start with "From " in email bodies are prefixed with - a '>' character by generators. The default is ``True`` for - :attr:`~email.policy.compat32` and ``False`` for all other policies. - (Contributed by Milan Oberkirch in :issue:`20098`.) +A new policy option :attr:`Policy.mangle_from_ <email.policy.Policy.mangle_from_>` +controls whether or not lines that start with ``"From "`` in email bodies are +prefixed with a ``">"`` character by generators. The default is ``True`` for +:attr:`~email.policy.compat32` and ``False`` for all other policies. +(Contributed by Milan Oberkirch in :issue:`20098`.) + +A new +:meth:`Message.get_content_disposition <email.message.Message.get_content_disposition>` +method provides easy access to a canonical value for the +:mailheader:`Content-Disposition` header. +(Contributed by Abhilash Raj in :issue:`21083`.) + +A new policy option :attr:`EmailPolicy.utf8 <email.policy.EmailPolicy.utf8>` +can be set to ``True`` to encode email headers using the UTF-8 charset instead +of using encoded words. This allows ``Messages`` to be formatted according to +:rfc:`6532` and used with an SMTP server that supports the :rfc:`6531` +``SMTPUTF8`` extension. (Contributed by R. David Murray in +:issue:`24211`.) + + +enum +---- + +The :class:`~enum.Enum` callable has a new parameter *start* to +specify the initial number of enum values if only *names* are provided:: + + >>> Animal = enum.Enum('Animal', 'cat dog', start=10) + >>> Animal.cat + <Animal.cat: 10> + >>> Animal.dog + <Animal.dog: 11> + +(Contributed by Ethan Furman in :issue:`21706`.) + + +faulthandler +------------ -* A new method :meth:`~email.message.Message.get_content_disposition` provides - easy access to a canonical value for the :mailheader:`Content-Disposition` - header (``None`` if there is no such header). (Contributed by Abhilash Raj - in :issue:`21083`.) +:func:`~faulthandler.enable`, :func:`~faulthandler.register`, +:func:`~faulthandler.dump_traceback` and +:func:`~faulthandler.dump_traceback_later` functions now accept file +descriptors in addition to file-like objects. +(Contributed by Wei Wu in :issue:`23566`.) + + +functools +--------- + +Most of :func:`~functools.lru_cache` machinery is now implemented in C, making +it significantly faster. (Contributed by Matt Joiner, Alexey Kachayev, and +Serhiy Storchaka in :issue:`14373`.) -* A new policy option :attr:`~email.policy.EmailPolicy.utf8` can be set - ``True`` to encode email headers using the utf8 charset instead of using - encoded words. This allows ``Messages`` to be formatted according to - :rfc:`6532` and used with an SMTP server that supports the :rfc:`6531` - ``SMTPUTF8`` extension. (Contributed by R. David Murray in :issue:`24211`.) glob ---- -* :func:`~glob.iglob` and :func:`~glob.glob` now support recursive search in - subdirectories using the "``**``" pattern. - (Contributed by Serhiy Storchaka in :issue:`13968`.) +:func:`~glob.iglob` and :func:`~glob.glob` functions now support recursive +search in subdirectories using the ``"**"`` pattern. +(Contributed by Serhiy Storchaka in :issue:`13968`.) + + +heapq +----- + +Element comparison in :func:`~heapq.merge` can now be customized by +passing a :term:`key function` in a new optional ``key`` keyword argument. +A new optional ``reverse`` keyword argument can be used to reverse element +comparison. (Contributed by Raymond Hettinger in :issue:`13742`.) + + +http +---- + +A new :class:`HTTPStatus <http.HTTPStatus>` enum that defines a set of +HTTP status codes, reason phrases and long descriptions written in English. +(Contributed by Demian Brecht in :issue:`21793`.) + idlelib and IDLE ---------------- Since idlelib implements the IDLE shell and editor and is not intended for -import by other programs, it gets improvements with every release. See +import by other programs, it gets improvements with every release. See :file:`Lib/idlelib/NEWS.txt` for a cumulative list of changes since 3.4.0, as well as changes made in future 3.5.x releases. This file is also available from the IDLE Help -> About Idle dialog. + imaplib ------- -* :class:`IMAP4` now supports the context management protocol. When used in a - :keyword:`with` statement, the IMAP4 ``LOGOUT`` command will be called - automatically at the end of the block. (Contributed by Tarek Ziadé and - Serhiy Storchaka in :issue:`4972`.) +The :class:`~imaplib.IMAP4` class now supports context manager protocol. +When used in a :keyword:`with` statement, the IMAP4 ``LOGOUT`` +command will be called automatically at the end of the block. +(Contributed by Tarek Ziadé and Serhiy Storchaka in :issue:`4972`.) + +The :mod:`imaplib` module now supports :rfc:`5161` (ENABLE Extension) +and :rfc:`6855` (UTF-8 Support) via the :meth:`IMAP4.enable <imaplib.IMAP4.enable>` +method. A new :attr:`IMAP4.utf8_enabled <imaplib.IMAP4.utf8_enabled>` +attribute, tracks whether or not :rfc:`6855` support is enabled. +(Contributed by Milan Oberkirch, R. David Murray, and Maciej Szulik in +:issue:`21800`.) -* :mod:`imaplib` now supports :rfc:`5161`: the :meth:`~imaplib.IMAP4.enable` - extension), and :rfc:`6855`: utf-8 support (internationalized email, via the - ``UTF8=ACCEPT`` argument to :meth:`~imaplib.IMAP4.enable`). A new attribute, - :attr:`~imaplib.IMAP4.utf8_enabled`, tracks whether or not :rfc:`6855` - support is enabled. Milan Oberkirch, R. David Murray, and Maciej Szulik in - :issue:`21800`.) +The :mod:`imaplib` module now automatically encodes non-ASCII string usernames +and passwords using UTF-8, as recommended by the RFCs. (Contributed by Milan +Oberkirch in :issue:`21800`.) -* :mod:`imaplib` now automatically encodes non-ASCII string usernames and - passwords using ``UTF8``, as recommended by the RFCs. (Contributed by Milan - Oberkirch in :issue:`21800`.) imghdr ------ -* :func:`~imghdr.what` now recognizes the `OpenEXR <http://www.openexr.com>`_ - format. (Contributed by Martin Vignali and Claudiu Popa in :issue:`20295`.) +The :func:`~imghdr.what` function now recognizes the +`OpenEXR <http://www.openexr.com>`_ format +(contributed by Martin Vignali and Claudiu Popa in :issue:`20295`), +and the `WebP <https://en.wikipedia.org/wiki/WebP>`_ format +(contributed by Fabrice Aneche and Claudiu Popa in :issue:`20197`.) + importlib --------- -* :class:`importlib.util.LazyLoader` allows for the lazy loading of modules in - applications where startup time is paramount. - (Contributed by Brett Cannon in :issue:`17621`.) +The :class:`util.LazyLoader <importlib.util.LazyLoader>` class allows for +lazy loading of modules in applications where startup time is important. +(Contributed by Brett Cannon in :issue:`17621`.) + +The :func:`abc.InspectLoader.source_to_code <importlib.abc.InspectLoader.source_to_code>` +method is now a static method. This makes it easier to initialize a module +object with code compiled from a string by running +``exec(code, module.__dict__)``. +(Contributed by Brett Cannon in :issue:`21156`.) -* :func:`importlib.abc.InspectLoader.source_to_code` is now a - static method to make it easier to work with source code in a string. - With a module object that you want to initialize you can then use - ``exec(code, module.__dict__)`` to execute the code in the module. +The new :func:`util.module_from_spec <importlib.util.module_from_spec>` +function is now the preferred way to create a new module. As opposed to +creating a :class:`types.ModuleType` instance directly, this new function +will set the various import-controlled attributes based on the passed-in +spec object. (Contributed by Brett Cannon in :issue:`20383`.) -* :func:`importlib.util.module_from_spec` is now the preferred way to create a - new module. Compared to :class:`types.ModuleType`, this new function will set - the various import-controlled attributes based on the passed-in spec object. inspect ------- -* :class:`inspect.Signature` and :class:`inspect.Parameter` are now - picklable and hashable. (Contributed by Yury Selivanov in :issue:`20726` - and :issue:`20334`.) +Both :class:`~inspect.Signature` and :class:`~inspect.Parameter` classes are +now picklable and hashable. (Contributed by Yury Selivanov in :issue:`20726` +and :issue:`20334`.) + +A new +:meth:`BoundArguments.apply_defaults <inspect.BoundArguments.apply_defaults>` +method provides a way to set default values for missing arguments. +(Contributed by Yury Selivanov in :issue:`24190`.) -* New method :meth:`inspect.BoundArguments.apply_defaults`. (Contributed - by Yury Selivanov in :issue:`24190`.) +A new class method +:meth:`Signature.from_callable <inspect.Signature.from_callable>` makes +subclassing of :class:`~inspect.Signature` easier. (Contributed +by Yury Selivanov and Eric Snow in :issue:`17373`.) -* New class method :meth:`inspect.Signature.from_callable`, which makes - subclassing of :class:`~inspect.Signature` easier. (Contributed - by Yury Selivanov and Eric Snow in :issue:`17373`.) +The :func:`~inspect.signature` function now accepts a ``follow_wrapped`` +optional keyword argument, which, when set to ``False``, disables automatic +following of ``__wrapped__`` links. +(Contributed by Yury Selivanov in :issue:`20691`.) -* New argument ``follow_wrapped`` for :func:`inspect.signature`. - (Contributed by Yury Selivanov in :issue:`20691`.) +A set of new functions to inspect +:term:`coroutine functions <coroutine function>` and +:term:`coroutine objects <coroutine>` has been added: +:func:`~inspect.iscoroutine`, :func:`~inspect.iscoroutinefunction`, +:func:`~inspect.isawaitable`, :func:`~inspect.getcoroutinelocals`, +and :func:`~inspect.getcoroutinestate`. +(Contributed by Yury Selivanov in :issue:`24017` and :issue:`24400`.) -* New :func:`~inspect.iscoroutine`, :func:`~inspect.iscoroutinefunction` - and :func:`~inspect.isawaitable` functions. (Contributed by - Yury Selivanov in :issue:`24017`.) +:func:`~inspect.stack`, :func:`~inspect.trace`, +:func:`~inspect.getouterframes`, and :func:`~inspect.getinnerframes` +functions now return a list of named tuples. +(Contributed by Daniel Shahaf in :issue:`16808`.) + + +io +-- + +A new :meth:`BufferedIOBase.readinto1 <io.BufferedIOBase.readinto1>` +method, that uses at most one call to the underlying raw stream's +:meth:`RawIOBase.read <io.RawIOBase.read>` (or +:meth:`RawIOBase.readinto <io.RawIOBase.readinto>`) method. +(Contributed by Nikolaus Rath in :issue:`20578`.) -* New :func:`~inspect.getcoroutinelocals` and :func:`~inspect.getcoroutinestate` - functions. (Contributed by Yury Selivanov in :issue:`24400`.) ipaddress --------- -* :class:`ipaddress.IPv4Network` and :class:`ipaddress.IPv6Network` now - accept an ``(address, netmask)`` tuple argument, so as to easily construct - network objects from existing addresses. (Contributed by Peter Moody - and Antoine Pitrou in :issue:`16531`.) +Both :class:`~ipaddress.IPv4Network` and :class:`~ipaddress.IPv6Network` classes +now accept an ``(address, netmask)`` tuple argument, so as to easily construct +network objects from existing addresses. (Contributed by Peter Moody +and Antoine Pitrou in :issue:`16531`.) + +A new :attr:`~ipaddress.IPv4Network.reverse_pointer>` attribute for +:class:`~ipaddress.IPv4Network` and :class:`~ipaddress.IPv6Network` classes +returns the name of the reverse DNS PTR record. +(Contributed by Leon Weber in :issue:`20480`.) + json ---- -* The output of :mod:`json.tool` command line interface is now in the same - order as the input. Use the :option:`--sort-keys` option to sort the output - of dictionaries alphabetically by key. (Contributed by Berker Peksag in - :issue:`21650`.) +The :mod:`json.tool` command line interface now preserves the order of keys in +JSON objects passed in input. The new ``--sort-keys`` option can be used +to sort the keys alphabetically. (Contributed by Berker Peksag +in :issue:`21650`.) + +JSON decoder now raises :exc:`json.JSONDecodeError` instead of +:exc:`ValueError`. (Contributed by Serhiy Storchaka in :issue:`19361`.) + + +locale +------ + +A new :func:`~locale.delocalize` function can be used to convert a string into +a normalized number string, taking the ``LC_NUMERIC`` settings into account. +(Contributed by Cédric Krier in :issue:`13918`.) + + +logging +------- + +All logging methods (:class:`~logging.Logger` :meth:`~logging.Logger.log`, +:meth:`~logging.Logger.exception`, :meth:`~logging.Logger.critical`, +:meth:`~logging.Logger.debug`, etc.), now accept exception instances +as an ``exc_info`` argument, in addition to boolean values and exception +tuples. (Contributed by Yury Selivanov in :issue:`20537`.) + +The :class:`handlers.HTTPHandler <logging.handlers.HTTPHandler>` class now +accepts an optional :class:`ssl.SSLContext` instance to configure SSL +settings used in an HTTP connection. +(Contributed by Alex Gaynor in :issue:`22788`.) + +The :class:`handlers.QueueListener <logging.handlers.QueueListener>` class now +takes a *respect_handler_level* keyword argument which, if set to ``True``, +will pass messages to handlers taking handler levels into account. +(Contributed by Vinay Sajip.) + + +lzma +---- + +The :meth:`LZMADecompressor.decompress <lzma.LZMADecompressor.decompress>` +method now accepts an optional *max_length* argument to limit the maximum +size of decompressed data. +(Contributed by Martin Panter in :issue:`15955`.) -* JSON decoder now raises :exc:`json.JSONDecodeError` instead of - :exc:`ValueError`. (Contributed by Serhiy Storchaka in :issue:`19361`.) math ---- -* :data:`math.inf` and :data:`math.nan` constants added. (Contributed by Mark - Dickinson in :issue:`23185`.) -* :func:`math.isclose` function added. - (Contributed by Chris Barker and Tal Einat in :issue:`24270`.) +Two new constants have been added to the :mod:`math` module: :data:`~math.inf` +and :data:`~math.nan`. (Contributed by Mark Dickinson in :issue:`23185`.) + +A new function :func:`~math.isclose` provides a way to test for approximate +equality. (Contributed by Chris Barker and Tal Einat in :issue:`24270`.) + +A new :func:`~math.gcd` function has been added. The :func:`fractions.gcd` +function is now deprecated. (Contributed by Mark Dickinson and Serhiy +Storchaka in :issue:`22486`.) + + +operator +-------- + +:func:`~operator.attrgetter`, :func:`~operator.itemgetter`, +and :func:`~operator.methodcaller` objects now support pickling. +(Contributed by Josh Rosenberg and Serhiy Storchaka in :issue:`22955`.) + +New :func:`~operator.matmul` and :func:`~operator.imatmul` functions +to perform matrix multiplication. +(Contributed by Benjamin Peterson in :issue:`21176`.) + os -- -* New :func:`os.scandir` function that exposes file information from - the operating system when listing a directory. :func:`os.scandir` - returns an iterator of :class:`os.DirEntry` objects corresponding to - the entries in the directory given by *path*. (Contributed by Ben - Hoyt with the help of Victor Stinner in :issue:`22524`.) +The new :func:`~os.scandir` function returning an iterator of +:class:`~os.DirEntry` objects has been added. If possible, :func:`~os.scandir` +extracts file attributes while scanning a directory, removing the need to +perform subsequent system calls to determine file type or attributes, which may +significantly improve performance. (Contributed by Ben Hoyt with the help +of Victor Stinner in :issue:`22524`.) + +On Windows, a new +:attr:`stat_result.st_file_attributes <os.stat_result.st_file_attributes>` +attribute is now available. It corresponds to ``dwFileAttributes`` member of +the ``BY_HANDLE_FILE_INFORMATION`` structure returned by +``GetFileInformationByHandle()``. (Contributed by Ben Hoyt in :issue:`21719`.) + +The :func:`~os.urandom` function now uses ``getrandom()`` syscall on Linux 3.17 +or newer, and ``getentropy()`` on OpenBSD 5.6 and newer, removing the need to +use ``/dev/urandom`` and avoiding failures due to potential file descriptor +exhaustion. (Contributed by Victor Stinner in :issue:`22181`.) + +New :func:`~os.get_blocking` and :func:`~os.set_blocking` functions allow to +get and set a file descriptor blocking mode (:data:`~os.O_NONBLOCK`.) +(Contributed by Victor Stinner in :issue:`22054`.) + +The :func:`~os.truncate` and :func:`~os.ftruncate` functions are now supported +on Windows. (Contributed by Steve Dower in :issue:`23668`.) -* :class:`os.stat_result` now has a :attr:`~os.stat_result.st_file_attributes` - attribute on Windows. (Contributed by Ben Hoyt in :issue:`21719`.) +There is a new :func:`os.path.commonpath` function returning the longest +common sub-path of each passed pathname. Unlike the +:func:`os.path.commonprefix` function, it always returns a valid +path. (Contributed by Rafik Draoui and Serhiy Storchaka in :issue:`10395`.) -* :func:`os.urandom`: On Linux 3.17 and newer, the ``getrandom()`` syscall is - now used when available. On OpenBSD 5.6 and newer, the C ``getentropy()`` - function is now used. These functions avoid the usage of an internal file - descriptor. -os.path +pathlib ------- -* New :func:`~os.path.commonpath` function that extracts common path prefix. - Unlike the :func:`~os.path.commonprefix` function, it always returns a valid - path. (Contributed by Rafik Draoui and Serhiy Storchaka in :issue:`10395`.) +The new :meth:`Path.samefile <pathlib.Path.samefile>` method can be used +to check whether the path points to the same file as other path, which can be +either an another :class:`~pathlib.Path` object, or a string. +(Contributed by Vajrasky Kok and Antoine Pitrou in :issue:`19775`.) + +The :meth:`Path.mkdir <pathlib.Path.mkdir>` method how accepts a new optional +``exist_ok`` argument to match ``mkdir -p`` and :func:`os.makrdirs` +functionality. (Contributed by Berker Peksag in :issue:`21539`.) + +There is a new :meth:`Path.expanduser <pathlib.Path.expanduser>` method to +expand ``~`` and ``~user`` prefixes. (Contributed by Serhiy Storchaka and +Claudiu Popa in :issue:`19776`.) + +A new :meth:`Path.home <pathlib.Path.home>` class method can be used to get +an instance of :class:`~pathlib.Path` object representing the user’s home +directory. +(Contributed by Victor Salgado and Mayank Tripathi in :issue:`19777`.) + +New :meth:`Path.write_text <pathlib.Path.write_text>`, +:meth:`Path.read_text <pathlib.Path.read_text>`, +:meth:`Path.write_bytes <pathlib.Path.write_bytes>`, +:meth:`Path.read_bytes <pathlib.Path.read_bytes>` methods to simplify +read/write operations on files. +(Contributed by Christopher Welborn in :issue:`20218`.) + pickle ------ -* Serializing more "lookupable" objects (such as unbound methods or nested - classes) now are supported with pickle protocols < 4. - (Contributed by Serhiy Storchaka in :issue:`23611`.) +Nested objects, such as unbound methods or nested classes, can now be pickled +using :ref:`pickle protocols <pickle-protocols>` older than protocol version 4. +Protocol version 4 already supports these cases. (Contributed by Serhiy +Storchaka in :issue:`23611`.) + poplib ------ -* A new command :meth:`~poplib.POP3.utf8` enables :rfc:`6856` - (internationalized email) support if the POP server supports it. (Contributed - by Milan OberKirch in :issue:`21804`.) +A new :meth:`POP3.utf8 <poplib.POP3.utf8>` command enables :rfc:`6856` +(Internationalized Email) support, if a POP server supports it. +(Contributed by Milan OberKirch in :issue:`21804`.) + re -- -* Number of capturing groups in regular expression is no longer limited by 100. - (Contributed by Serhiy Storchaka in :issue:`22437`.) +The number of capturing groups in regular expression is no longer limited by +100. (Contributed by Serhiy Storchaka in :issue:`22437`.) + +The :func:`~re.sub` and :func:`~re.subn` functions now replace unmatched +groups with empty strings instead of raising an exception. +(Contributed by Serhiy Storchaka in :issue:`1519638`.) + +The :class:`re.error` exceptions have new attributes: +:attr:`~re.error.msg`, :attr:`~re.error.pattern`, +:attr:`~re.error.pos`, :attr:`~re.error.lineno`, +and :attr:`~re.error.colno` that provide better context +information about the error. +(Contributed by Serhiy Storchaka in :issue:`22578`.) + + +readline +-------- + +A new :func:`~readline.append_history_file` function can be used to append +the specified number of trailing elements in history to the given file. +(Contributed by Bruno Cauet in :issue:`22940`.) + + +selectors +--------- + +The new :class:`~selectors.DevpollSelector` supports efficient +``/dev/poll`` polling on Solaris. +(Contributed by Giampaolo Rodola' in :issue:`18931`.) -* Now unmatched groups are replaced with empty strings in :func:`re.sub` - and :func:`re.subn`. (Contributed by Serhiy Storchaka in :issue:`1519638`.) shutil ------ -* :func:`~shutil.move` now accepts a *copy_function* argument, allowing, - for example, :func:`~shutil.copy` to be used instead of the default - :func:`~shutil.copy2` if there is a need to ignore metadata. (Contributed by - Claudiu Popa in :issue:`19840`.) +The :func:`~shutil.move` function now accepts a *copy_function* argument, +allowing, for example, the :func:`~shutil.copy` function to be used instead of +the default :func:`~shutil.copy2` if there is a need to ignore file metadata +when moving. +(Contributed by Claudiu Popa in :issue:`19840`.) + +The :func:`~shutil.make_archive` function now supports the *xztar* format. +(Contributed by Serhiy Storchaka in :issue:`5411`.) + signal ------ -* On Windows, :func:`signal.set_wakeup_fd` now also supports socket handles. - (Contributed by Victor Stinner in :issue:`22018`.) +On Windows, the :func:`~signal.set_wakeup_fd` function now also supports +socket handles. (Contributed by Victor Stinner in :issue:`22018`.) + +Various ``SIG*`` constants in the :mod:`signal` module have been converted into +:mod:`Enums <enum>`. This allows meaningful names to be printed +during debugging, instead of integer "magic numbers". +(Contributed by Giampaolo Rodola' in :issue:`21076`.) -* Different constants of :mod:`signal` module are now enumeration values using - the :mod:`enum` module. This allows meaningful names to be printed during - debugging, instead of integer “magic numbers”. (Contributed by Giampaolo - Rodola' in :issue:`21076`.) smtpd ----- -* Both :class:`~smtpd.SMTPServer` and :class:`smtpd.SMTPChannel` now accept a - *decode_data* keyword to determine if the DATA portion of the SMTP - transaction is decoded using the ``utf-8`` codec or is instead provided to - :meth:`~smtpd.SMTPServer.process_message` as a byte string. The default - is ``True`` for backward compatibility reasons, but will change to ``False`` - in Python 3.6. If *decode_data* is set to ``False``, the - :meth:`~smtpd.SMTPServer.process_message` method must be prepared to accept - keyword arguments. (Contributed by Maciej Szulik in :issue:`19662`.) - -* :class:`~smtpd.SMTPServer` now advertises the ``8BITMIME`` extension - (:rfc:`6152`) if if *decode_data* has been set ``True``. If the client - specifies ``BODY=8BITMIME`` on the ``MAIL`` command, it is passed to - :meth:`~smtpd.SMTPServer.process_message` via the ``mail_options`` keyword. - (Contributed by Milan Oberkirch and R. David Murray in :issue:`21795`.) - -* :class:`~smtpd.SMTPServer` now supports the ``SMTPUTF8`` extension - (:rfc:`6531`: Internationalized Email). If the client specified ``SMTPUTF8 - BODY=8BITMIME`` on the ``MAIL`` command, they are passed to - :meth:`~smtpd.SMTPServer.process_message` via the ``mail_options`` keyword. - It is the responsibility of the :meth:`~smtpd.SMTPServer.process_message` - method to correctly handle the ``SMTPUTF8`` data. (Contributed by Milan - Oberkirch in :issue:`21725`.) - -* It is now possible to provide, directly or via name resolution, IPv6 - addresses in the :class:`~smtpd.SMTPServer` constructor, and have it - successfully connect. (Contributed by Milan Oberkirch in :issue:`14758`.) +Both :class:`~smtpd.SMTPServer` and :class:`~smtpd.SMTPChannel` classes now +accept a *decode_data* keyword argument to determine if the ``DATA`` portion of +the SMTP transaction is decoded using the ``"utf-8"`` codec or is instead +provided to the +:meth:`SMTPServer.process_message <smtpd.SMTPServer.process_message>` +method as a byte string. The default is ``True`` for backward compatibility +reasons, but will change to ``False`` in Python 3.6. If *decode_data* is set +to ``False``, the :meth:`~smtpd.SMTPServer.process_message` method must +be prepared to accept keyword arguments. +(Contributed by Maciej Szulik in :issue:`19662`.) + +The :class:`~smtpd.SMTPServer` class now advertises the ``8BITMIME`` extension +(:rfc:`6152`) if *decode_data* has been set ``True``. If the client +specifies ``BODY=8BITMIME`` on the ``MAIL`` command, it is passed to +:meth:`SMTPServer.process_message <smtpd.SMTPServer.process_message>` +via the ``mail_options`` keyword. +(Contributed by Milan Oberkirch and R. David Murray in :issue:`21795`.) + +The :class:`~smtpd.SMTPServer` class now also supports the ``SMTPUTF8`` +extension (:rfc:`6531`: Internationalized Email). If the client specified +``SMTPUTF8 BODY=8BITMIME`` on the ``MAIL`` command, they are passed to +:meth:`SMTPServer.process_message <smtpd.SMTPServer.process_message>` +via the ``mail_options`` keyword. It is the responsibility of the +:meth:`~smtpd.SMTPServer.process_message` method to correctly handle the +``SMTPUTF8`` data. (Contributed by Milan Oberkirch in :issue:`21725`.) + +It is now possible to provide, directly or via name resolution, IPv6 +addresses in the :class:`~smtpd.SMTPServer` constructor, and have it +successfully connect. (Contributed by Milan Oberkirch in :issue:`14758`.) + smtplib ------- -* A new :meth:`~smtplib.SMTP.auth` method provides a convenient way to - implement custom authentication mechanisms. - (Contributed by Milan Oberkirch in :issue:`15014`.) +A new :meth:`SMTP.auth <smtplib.SMTP.auth>` method provides a convenient way to +implement custom authentication mechanisms. (Contributed by Milan +Oberkirch in :issue:`15014`.) -* Additional debuglevel (2) shows timestamps for debug messages in - :class:`smtplib.SMTP`. (Contributed by Gavin Chappell and Maciej Szulik in - :issue:`16914`.) +The :meth:`SMTP.set_debuglevel <smtplib.SMTP.set_debuglevel>` method now +accepts an additional debuglevel (2), which enables timestamps in debug +messages. (Contributed by Gavin Chappell and Maciej Szulik in :issue:`16914`.) + +Both :meth:`SMTP.sendmail <smtplib.SMTP.sendmail>` and +:meth:`SMTP.send_message <smtplib.SMTP.send_message>` methods now +support support :rfc:`6531` (SMTPUTF8). +(Contributed by Milan Oberkirch and R. David Murray in :issue:`22027`.) -* :mod:`smtplib` now supports :rfc:`6531` (SMTPUTF8) in both the - :meth:`~smtplib.SMTP.sendmail` and :meth:`~smtplib.SMTP.send_message` - commands. (Contributed by Milan Oberkirch and R. David Murray in - :issue:`22027`.) sndhdr ------ -* :func:`~sndhdr.what` and :func:`~sndhdr.whathdr` now return - :func:`~collections.namedtuple`. - (Contributed by Claudiu Popa in :issue:`18615`.) +:func:`~sndhdr.what` and :func:`~sndhdr.whathdr` functions now return +a :func:`~collections.namedtuple`. (Contributed by Claudiu Popa in +:issue:`18615`.) + ssl --- -* The :meth:`~ssl.SSLSocket.do_handshake`, :meth:`~ssl.SSLSocket.read`, - :meth:`~ssl.SSLSocket.shutdown`, and :meth:`~ssl.SSLSocket.write` methods of - :class:`ssl.SSLSocket` don't reset the socket timeout anymore each time bytes - are received or sent. The socket timeout is now the maximum total duration of - the method. +Memory BIO Support +~~~~~~~~~~~~~~~~~~ + +(Contributed by Geert Jansen in :issue:`21965`.) + +The new :class:`~ssl.SSLObject` class has been added to provide SSL protocol +support for cases when the network I/O capabilities of :class:`~ssl.SSLSocket` +are not necessary or suboptimal. :class:`~ssl.SSLObject` represents +an SSL protocol instance, but does not implement any network I/O methods, and +instead provides a memory buffer interface. The new :class:`~ssl.MemoryBIO` +class can be used to pass data between Python and an SSL protocol instance. + +The memory BIO SSL support is primarily intended to be used in frameworks +implementing asynchronous I/O for which :class:`~ssl.SSLSocket`'s readiness +model ("select/poll") is inefficient. + +A new :meth:`SSLContext.wrap_bio <ssl.SSLContext.wrap_bio>` method can be used +to create a new :class:`~ssl.SSLObject` instance. + + +Application-Layer Protocol Negotiation Support +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +(Contributed by Benjamin Peterson in :issue:`20188`.) + +Where OpenSSL support is present, :mod:`ssl` module now implements +*Application-Layer Protocol Negotiation* TLS extension as described +in :rfc:`7301`. + +The new :meth:`SSLContext.set_alpn_protocols <ssl.SSLContext.set_alpn_protocols>` +can be used to specify which protocols a socket should advertise during +the TLS handshake. + +The new +:meth:`SSLSocket.selected_alpn_protocol <ssl.SSLSocket.selected_alpn_protocol>` +returns the protocol that was selected during the TLS handshake. +:data:`~ssl.HAS_ALPN` flag indicates whether APLN support is present. + + +Other Changes +~~~~~~~~~~~~~ + +There is a new :meth:`SSLSocket.version <ssl.SSLSocket.version>` method to query +the actual protocol version in use. +(Contributed by Antoine Pitrou in :issue:`20421`.) + +The :class:`~ssl.SSLSocket` class now implements +a :meth:`SSLSocket.sendfile <ssl.SSLSocket.sendfile>` method. +(Contributed by Giampaolo Rodola' in :issue:`17552`.) + +The :meth:`SSLSocket.send <ssl.SSLSocket.send>` method now raises either +:exc:`ssl.SSLWantReadError` or :exc:`ssl.SSLWantWriteError` exception on a +non-blocking socket if the operation would block. Previously, it would return +``0``. (Contributed by Nikolaus Rath in :issue:`20951`.) + +The :func:`~ssl.cert_time_to_seconds` function now interprets the input time +as UTC and not as local time, per :rfc:`5280`. Additionally, the return +value is always an :class:`int`. (Contributed by Akira Li in :issue:`19940`.) + +New :meth:`SSLObject.shared_ciphers <ssl.SSLObject.shared_ciphers>` and +:meth:`SSLSocket.shared_ciphers <ssl.SSLSocket.shared_ciphers>` methods return +the list of ciphers sent by the client during the handshake. +(Contributed by Benjamin Peterson in :issue:`23186`.) + +The :meth:`SSLSocket.do_handshake <ssl.SSLSocket.do_handshake>`, +:meth:`SSLSocket.read <ssl.SSLSocket.read>`, +:meth:`SSLSocket.shutdown <ssl.SSLSocket.shutdown>`, and +:meth:`SSLSocket.write <ssl.SSLSocket.write>` methods of :class:`ssl.SSLSocket` +class no longer reset the socket timeout every time bytes are received or sent. +The socket timeout is now the maximum total duration of the method. +(Contributed by Victor Stinner in :issue:`23853`.) + +The :func:`~ssl.match_hostname` function now supports matching of IP addresses. +(Contributed by Antoine Pitrou in :issue:`23239`.) -* Memory BIO Support: new classes :class:`~ssl.SSLObject`, - :class:`~ssl.MemoryBIO`, and new - :meth:`SSLContext.wrap_bio <ssl.SSLContext.wrap_bio>` method. - (Contributed by Geert Jansen in :issue:`21965`.) socket ------ -* New :meth:`socket.socket.sendfile` method allows to send a file over a socket - by using high-performance :func:`os.sendfile` function on UNIX resulting in - uploads being from 2x to 3x faster than when using plain - :meth:`socket.socket.send`. - (Contributed by Giampaolo Rodola' in :issue:`17552`.) +Functions with timeouts now use a monotonic clock, instead of a system clock. +(Contributed by Victor Stinner in :issue:`22043`.) + +A new :meth:`socket.sendfile <socket.socket.sendfile>` method allows to +send a file over a socket by using the high-performance :func:`os.sendfile` +function on UNIX resulting in uploads being from 2 to 3 times faster than when +using plain :meth:`socket.send <socket.socket.send>`. +(Contributed by Giampaolo Rodola' in :issue:`17552`.) + +The :meth:`socket.sendall <socket.socket.sendall>` method no longer resets the +socket timeout every time bytes are received or sent. The socket timeout is +now the maximum total duration to send all data. +(Contributed by Victor Stinner in :issue:`23853`.) + +The *backlog* argument of the :meth:`socket.listen <socket.socket.listen>` +method is now optional. By default it is set to +:data:`SOMAXCONN <socket.SOMAXCONN>` or to ``128`` whichever is less. +(Contributed by Charles-François Natali in :issue:`21455`.) + + +sqlite3 +------- + +The :class:`~sqlite3.Row` class now fully supports sequence protocol, +in particular :func:`reversed` iteration and slice indexing. +(Contributed by Claudiu Popa in :issue:`10203`; by Lucas Sinclair, +Jessica McKellar, and Serhiy Storchaka in :issue:`13583`.) -* The :meth:`socket.socket.sendall` method don't reset the socket timeout - anymore each time bytes are received or sent. The socket timeout is now the - maximum total duration to send all data. subprocess ---------- -* The new :func:`subprocess.run` function runs subprocesses and returns a - :class:`subprocess.CompletedProcess` object. It Provides a more consistent - API than :func:`~subprocess.call`, :func:`~subprocess.check_call` and - :func:`~subprocess.check_output`. +The new :func:`~subprocess.run` function has been added. +It runs the specified command and and returns a +:class:`~subprocess.CompletedProcess` object, which describes a finished +process. The new API is more consistent and is the recommended approach +to invoking subprocesses in Python code that does not need to maintain +compatibility with earlier Python versions. +(Contributed by Thomas Kluyver in :issue:`23342`.) + sys --- -* New :func:`~sys.set_coroutine_wrapper` and :func:`~sys.get_coroutine_wrapper` - functions. (Contributed by Yury Selivanov in :issue:`24017`.) +A new :func:`~sys.set_coroutine_wrapper` function allows setting a global +hook that will be called whenever a :term:`coroutine object <coroutine>` +is created by an :keyword:`async def` function. A corresponding +:func:`~sys.get_coroutine_wrapper` can be used to obtain a currently set +wrapper. Both functions are provisional, and are intended for debugging +purposes only. (Contributed by Yury Selivanov in :issue:`24017`.) + +A new :func:`~sys.is_finalizing` function can be used to check if the Python +interpreter is :term:`shutting down <interpreter shutdown>`. +(Contributed by Antoine Pitrou in :issue:`22696`.) + sysconfig --------- -* The user scripts directory on Windows is now versioned. - (Contributed by Paul Moore in :issue:`23437`.) +The name of the user scripts directory on Windows now includes the first +two components of Python version. (Contributed by Paul Moore +in :issue:`23437`.) + tarfile ------- -* The :func:`tarfile.open` function now supports ``'x'`` (exclusive creation) - mode. (Contributed by Berker Peksag in :issue:`21717`.) +The *mode* argument of the :func:`~tarfile.open` function now accepts ``"x"`` +to request exclusive creation. (Contributed by Berker Peksag in :issue:`21717`.) + +:meth:`TarFile.extractall <tarfile.TarFile.extractall>` and +:meth:`TarFile.extract <tarfile.TarFile.extract>` methods now take a keyword +argument *numeric_only*. If set to ``True``, the extracted files and +directories will be owned by the numeric ``uid`` and ``gid`` from the tarfile. +If set to ``False`` (the default, and the behavior in versions prior to 3.5), +they will be owned by the named user and group in the tarfile. +(Contributed by Michael Vogt and Eric Smith in :issue:`23193`.) + + +threading +--------- + +Both :meth:`Lock.acquire <threading.Lock.acquire>` and +:meth:`RLock.acquire <threading.RLock.acquire>` methods +now use a monotonic clock for timeout management. +(Contributed by Victor Stinner in :issue:`22043`.) -* The :meth:`~tarfile.TarFile.extractall` and :meth:`~tarfile.TarFile.extract` - methods now take a keyword parameter *numeric_only*. If set to ``True``, - the extracted files and directories will be owned by the numeric uid and gid - from the tarfile. If set to ``False`` (the default, and the behavior in - versions prior to 3.5), they will be owned bythe named user and group in the - tarfile. (Contributed by Michael Vogt and Eric Smith in :issue:`23193`.) time ---- -* The :func:`time.monotonic` function is now always available. (Contributed by - Victor Stinner in :issue:`22043`.) +The :func:`~time.monotonic` function is now always available. +(Contributed by Victor Stinner in :issue:`22043`.) + + +timeit +------ + +A new command line option ``-u`` or ``--unit=U`` can be used to specify the time +unit for the timer output. Supported options are ``usec``, ``msec``, +or ``sec``. (Contributed by Julian Gindi in :issue:`18983`.) + +The :func:`~timeit.timeit` function has a new *globals* parameter for +specifying the namespace in which the code will be running. +(Contributed by Ben Roberts in :issue:`2527`.) + tkinter ------- -* The :mod:`tkinter._fix` module used for setting up the Tcl/Tk environment - on Windows has been replaced by a private function in the :mod:`_tkinter` - module which makes no permanent changes to environment variables. - (Contributed by Zachary Ware in :issue:`20035`.) +The :mod:`tkinter._fix` module used for setting up the Tcl/Tk environment +on Windows has been replaced by a private function in the :mod:`_tkinter` +module which makes no permanent changes to environment variables. +(Contributed by Zachary Ware in :issue:`20035`.) + + +traceback +--------- + +New :func:`~traceback.walk_stack` and :func:`~traceback.walk_tb` +functions to conveniently traverse frame and traceback objects. +(Contributed by Robert Collins in :issue:`17911`.) + +New lightweight classes: :class:`~traceback.TracebackException`, +:class:`~traceback.StackSummary`, and :class:`traceback.FrameSummary`. +(Contributed by Robert Collins in :issue:`17911`.) + +Both :func:`~traceback.print_tb` and :func:`~traceback.print_stack` functions +now support negative values for the *limit* argument. +(Contributed by Dmitry Kazakov in :issue:`22619`.) + types ----- -* New :func:`~types.coroutine` function. (Contributed by Yury Selivanov - in :issue:`24017`.) +A new :func:`~types.coroutine` function to transform +:term:`generator <generator iterator>` and +:class:`generator-like <collections.abc.Generator>` objects into +:term:`awaitables <awaitable>`. +(Contributed by Yury Selivanov in :issue:`24017`.) + +A new :class:`~types.CoroutineType` is the type of :term:`coroutine` objects +created by :keyword:`async def` functions. +(Contributed by Yury Selivanov in :issue:`24400`.) -* New :class:`~types.CoroutineType`. (Contributed by Yury Selivanov - in :issue:`24400`.) urllib ------ -* A new :class:`~urllib.request.HTTPPasswordMgrWithPriorAuth` allows HTTP Basic - Authentication credentials to be managed so as to eliminate unnecessary - ``401`` response handling, or to unconditionally send credentials - on the first request in order to communicate with servers that return a - ``404`` response instead of a ``401`` if the ``Authorization`` header is not - sent. (Contributed by Matej Cepl in :issue:`19494` and Akshit Khurana in - :issue:`7159`.) +A new +:class:`request.HTTPPasswordMgrWithPriorAuth <urllib.request.HTTPPasswordMgrWithPriorAuth>` +class allows HTTP Basic Authentication credentials to be managed so as to +eliminate unnecessary ``401`` response handling, or to unconditionally send +credentials on the first request in order to communicate with servers that +return a ``404`` response instead of a ``401`` if the ``Authorization`` header +is not sent. (Contributed by Matej Cepl in :issue:`19494` and Akshit Khurana in +:issue:`7159`.) + +A new *quote_via* argument for the +:func:`parse.urlencode <urllib.parse.urlencode>` +function provides a way to control the encoding of query parts if needed. +(Contributed by Samwyse and Arnon Yaari in :issue:`13866`.) + +The :func:`request.urlopen <urllib.request.urlopen>` function accepts an +:class:`ssl.SSLContext` object as a *context* argument, which will be used for +the HTTPS connection. (Contributed by Alex Gaynor in :issue:`22366`.) + +The :func:`parse.urljoin <urllib.parse.urljoin>` was updated to use the +:rfc:`3986` semantics for the resolution of relative URLs, rather than +:rfc:`1808` and :rfc:`2396`. +(Contributed by Demian Brecht and Senthil Kumaran in :issue:`22118`.) -* A new :func:`~urllib.parse.urlencode` parameter *quote_via* provides a way to - control the encoding of query parts if needed. (Contributed by Samwyse and - Arnon Yaari in :issue:`13866`.) unicodedata ----------- -* The :mod:`unicodedata` module now uses data from `Unicode 8.0.0 - <http://unicode.org/versions/Unicode8.0.0/>`_. +The :mod:`unicodedata` module now uses data from `Unicode 8.0.0 +<http://unicode.org/versions/Unicode8.0.0/>`_. + + +unittest +-------- + +A new command line option ``--locals`` to show local variables in +tracebacks. (Contributed by Robert Collins in :issue:`22936`.) + + +unittest.mock +------------- + +The :class:`~unittest.mock.Mock` has the following improvements: + +* Class constructor has a new *unsafe* parameter, which causes mock + objects to raise :exc:`AttributeError` on attribute names starting + with ``"assert"``. + (Contributed by Kushal Das in :issue:`21238`.) + +* A new :meth:`Mock.assert_not_called <unittest.mock.Mock.assert_not_called>` + method to check if the mock object was called. + (Contributed by Kushal Das in :issue:`21262`.) + +The :class:`~unittest.mock.MagicMock` class now supports :meth:`__truediv__`, +:meth:`__divmod__` and :meth:`__matmul__` operators. +(Contributed by Johannes Baiter in :issue:`20968`, and Håkan Lövdahl +in :issue:`23581` and :issue:`23568`.) wsgiref ------- -* *headers* parameter of :class:`wsgiref.headers.Headers` is now optional. - (Contributed by Pablo Torres Navarrete and SilentGhost in :issue:`5800`.) +The *headers* argument of the :class:`headers.Headers <wsgiref.headers.Headers>` +class constructor is now optional. +(Contributed by Pablo Torres Navarrete and SilentGhost in :issue:`5800`.) + xmlrpc ------ -* :class:`xmlrpc.client.ServerProxy` is now a :term:`context manager`. - (Contributed by Claudiu Popa in :issue:`20627`.) +The :class:`client.ServerProxy <xmlrpc.client.ServerProxy>` class is now a +:term:`context manager`. +(Contributed by Claudiu Popa in :issue:`20627`.) + +:class:`client.ServerProxy <xmlrpc.client.ServerProxy>` constructor now accepts +an optional :class:`ssl.SSLContext` instance. +(Contributed by Alex Gaynor in :issue:`22960`.) + xml.sax ------- -* SAX parsers now support a character stream of - :class:`~xml.sax.xmlreader.InputSource` object. - (Contributed by Serhiy Storchaka in :issue:`2175`.) +SAX parsers now support a character stream of the +:class:`xmlreader.InputSource <xml.sax.xmlreader.InputSource>` object. +(Contributed by Serhiy Storchaka in :issue:`2175`.) -faulthandler ------------- - -* :func:`~faulthandler.enable`, :func:`~faulthandler.register`, - :func:`~faulthandler.dump_traceback` and - :func:`~faulthandler.dump_traceback_later` functions now accept file - descriptors. (Contributed by Wei Wu in :issue:`23566`.) zipfile ------- -* Added support for writing ZIP files to unseekable streams. - (Contributed by Serhiy Storchaka in :issue:`23252`.) +ZIP output can now be written to unseekable streams. +(Contributed by Serhiy Storchaka in :issue:`23252`.) + +The *mode* argument of :meth:`ZipFile.open <zipfile.ZipFile.open>` method now +accepts ``"x"`` to request exclusive creation. +(Contributed by Serhiy Storchaka in :issue:`21717`.) -* The :func:`zipfile.ZipFile.open` function now supports ``'x'`` (exclusive - creation) mode. (Contributed by Serhiy Storchaka in :issue:`21717`.) + +Other module-level changes +========================== + +Many functions in :mod:`mmap`, :mod:`ossaudiodev`, :mod:`socket`, +:mod:`ssl`, and :mod:`codecs` modules now accept writable +:term:`bytes-like objects <bytes-like object>`. +(Contributed by Serhiy Storchaka in :issue:`23001`.) Optimizations ============= -The following performance enhancements have been added: +The :func:`os.walk` function has been sped up by 3 to 5 times on POSIX systems, +and by 7 to 20 times on Windows. This was done using the new :func:`os.scandir` +function, which exposes file information from the underlying ``readdir`` or +``FindFirstFile``/``FindNextFile`` system calls. (Contributed by +Ben Hoyt with help from Victor Stinner in :issue:`23605`.) + +Construction of ``bytes(int)`` (filled by zero bytes) is faster and uses less +memory for large objects. ``calloc()`` is used instead of ``malloc()`` to +allocate memory for these objects. +(Contributed by Victor Stinner in :issue:`21233`.) + +Some operations on :mod:`ipaddress` :class:`~ipaddress.IPv4Network` and +:class:`~ipaddress.IPv6Network` have been massively sped up, such as +:meth:`~ipaddress.IPv4Network.subnets`, :meth:`~ipaddress.IPv4Network.supernet`, +:func:`~ipaddress.summarize_address_range`, :func:`~ipaddress.collapse_addresses`. +The speed up can range from 3 to 15 times. +(Contributed by Antoine Pitrou, Michel Albert, and Markus in +:issue:`21486`, :issue:`21487`, :issue:`20826`, :issue:`23266`.) + +Pickling of :mod:`ipaddress` objects was optimized to produce significantly +smaller output. (Contributed by Serhiy Storchaka in :issue:`23133`.) + +Many operations on :class:`io.BytesIO` are now 50% to 100% faster. +(Contributed by Serhiy Storchaka in :issue:`15381` and David Wilson in +:issue:`22003`.) + +The :func:`marshal.dumps` function is now faster: 65-85% with versions 3 +and 4, 20-25% with versions 0 to 2 on typical data, and up to 5 times in +best cases. +(Contributed by Serhiy Storchaka in :issue:`20416` and :issue:`23344`.) + +The UTF-32 encoder is now 3 to 7 times faster. +(Contributed by Serhiy Storchaka in :issue:`15027`.) + +Regular expressions are now parsed up to 10% faster. +(Contributed by Serhiy Storchaka in :issue:`19380`.) + +The :func:`json.dumps` function was optimized to run with +``ensure_ascii=False`` as fast as with ``ensure_ascii=True``. +(Contributed by Naoki Inada in :issue:`23206`.) -* :func:`os.walk` has been sped up by 3-5x on POSIX systems and 7-20x - on Windows. This was done using the new :func:`os.scandir` function, - which exposes file information from the underlying ``readdir`` and - ``FindFirstFile``/``FindNextFile`` system calls. (Contributed by - Ben Hoyt with help from Victor Stinner in :issue:`23605`.) +The :c:func:`PyObject_IsInstance` and :c:func:`PyObject_IsSubclass` +functions have been sped up in the common case that the second argument +has :class:`type` as its metaclass. +(Contributed Georg Brandl by in :issue:`22540`.) -* Construction of ``bytes(int)`` (filled by zero bytes) is faster and uses less - memory for large objects. ``calloc()`` is used instead of ``malloc()`` to - allocate memory for these objects. +Method caching was slightly improved, yielding up to 5% performance +improvement in some benchmarks. +(Contributed by Antoine Pitrou in :issue:`22847`.) -* Some operations on :class:`~ipaddress.IPv4Network` and - :class:`~ipaddress.IPv6Network` have been massively sped up, such as - :meth:`~ipaddress.IPv4Network.subnets`, :meth:`~ipaddress.IPv4Network.supernet`, - :func:`~ipaddress.summarize_address_range`, :func:`~ipaddress.collapse_addresses`. - The speed up can range from 3x to 15x. - (:issue:`21486`, :issue:`21487`, :issue:`20826`) +Objects from :mod:`random` module now use two times less memory on 64-bit +builds. (Contributed by Serhiy Storchaka in :issue:`23488`.) -* Many operations on :class:`io.BytesIO` are now 50% to 100% faster. - (Contributed by Serhiy Storchaka in :issue:`15381` and David Wilson in - :issue:`22003`.) +The :func:`property` getter calls are up to 25% faster. +(Contributed by Joe Jevnik in :issue:`23910`.) -* :func:`marshal.dumps` is now faster (65%-85% with versions 3--4, 20-25% with - versions 0--2 on typical data, and up to 5x in best cases). - (Contributed by Serhiy Storchaka in :issue:`20416` and :issue:`23344`.) +Instantiation of :class:`fractions.Fraction` is now up to 30% faster. +(Contributed by Stefan Behnel in :issue:`22464`.) -* The UTF-32 encoder is now 3x to 7x faster. (Contributed by Serhiy Storchaka - in :issue:`15027`.) +String methods :meth:`~str.find`, :meth:`~str.rfind`, :meth:`~str.split`, +:meth:`~str.partition` and :keyword:`in` string operator are now significantly +faster for searching 1-character substrings. +(Contributed by Serhiy Storchaka in :issue:`23573`.) Build and C API Changes ======================= -Changes to Python's build process and to the C API include: +New ``calloc`` functions were added: + + * :c:func:`PyMem_RawCalloc`, + * :c:func:`PyMem_Calloc`, + * :c:func:`PyObject_Calloc`, + * :c:func:`_PyObject_GC_Calloc`. + +(Contributed by Victor Stinner in :issue:`21233`.) + +New encoding/decoding helper functions: + + * :c:func:`Py_DecodeLocale` (replaced ``_Py_char2wchar()``), + * :c:func:`Py_EncodeLocale` (replaced ``_Py_wchar2char()``). + +(Contributed by Victor Stinner in :issue:`18395`.) + +A new :c:func:`PyCodec_NameReplaceErrors` function to replace the unicode +encode error with ``\N{...}`` escapes. +(Contributed by Serhiy Storchaka in :issue:`19676`.) + +A new :c:func:`PyErr_FormatV` function similar to :c:func:`PyErr_Format`, +but accepts a ``va_list`` argument. +(Contributed by Antoine Pitrou in :issue:`18711`.) + +A new :c:data:`PyExc_RecursionError` exception. +(Contributed by Georg Brandl in :issue:`19235`.) + +New :c:func:`PyModule_FromDefAndSpec`, :c:func:`PyModule_FromDefAndSpec2`, +and :c:func:`PyModule_ExecDef` introduced by :pep:`489` -- multi-phase +extension module initialization. +(Contributed by Petr Viktorin in :issue:`24268`.) + +New :c:func:`PyNumber_MatrixMultiply` and +:c:func:`PyNumber_InPlaceMatrixMultiply` functions to perform matrix +multiplication. +(Contributed by Benjamin Peterson in :issue:`21176`. See also :pep:`465` +for details.) -* New ``calloc`` functions: +The :c:member:`PyTypeObject.tp_finalize` slot is now part of stable ABI. - * :c:func:`PyMem_RawCalloc` - * :c:func:`PyMem_Calloc` - * :c:func:`PyObject_Calloc` - * :c:func:`_PyObject_GC_Calloc` +Windows builds now require Microsoft Visual C++ 14.0, which +is available as part of `Visual Studio 2015 <http://www.visualstudio.com>`_. + +Extension modules now include platform information tag in their filename on +some platforms (the tag is optional, and CPython will import extensions without +it; although if the tag is present and mismatched, the extension won't be +loaded): + +* On Linux, extension module filenames end with + ``.cpython-<major><minor>m-<architecture>-<os>.pyd``: + + * ``<major>`` is the major number of the Python version; + for Python 3.5 this is ``3``. + + * ``<minor>`` is the minor number of the Python version; + for Python 3.5 this is ``5``. + + * ``<architecture>`` is the hardware architecture the extension module + was built to run on. It's most commonly either ``i386`` for 32-bit Intel + platforms or ``x86_64`` for 64-bit Intel (and AMD) platforms. + + * ``<os>`` is always ``linux-gnu``, except for extensions built to + talk to the 32-bit ABI on 64-bit platforms, in which case it is + ``linux-gnu32`` (and ``<architecture>`` will be ``x86_64``). + +* On Windows, extension module filenames end with + ``<debug>.cp<major><minor>-<platform>.pyd``: + + * ``<major>`` is the major number of the Python version; + for Python 3.5 this is ``3``. + + * ``<minor>`` is the minor number of the Python version; + for Python 3.5 this is ``5``. + + * ``<platform>`` is the platform the extension module was built for, + either ``win32`` for Win32, ``win_amd64`` for Win64, ``win_ia64`` for + Windows Itanium 64, and ``win_arm`` for Windows on ARM. + + * If built in debug mode, ``<debug>`` will be ``_d``, + otherwise it will be blank. + +* On OS X platforms, extension module filenames now end with ``-darwin.so``. + +* On all other platforms, extension module filenames are the same as they were + with Python 3.4. -* Windows builds now require Microsoft Visual C++ 14.0, which - is available as part of `Visual Studio 2015 <http://www.visualstudio.com>`_. Deprecated ========== @@ -1031,63 +1779,59 @@ become proper keywords in Python 3.7. Unsupported Operating Systems ----------------------------- -* Windows XP - Per :PEP:`11`, Microsoft support of Windows XP has ended. +Windows XP is no longer supported by Microsoft, thus, per :PEP:`11`, CPython +3.5 is no longer officially supported on this OS. Deprecated Python modules, functions and methods ------------------------------------------------ -* The :mod:`formatter` module has now graduated to full deprecation and is still - slated for removal in Python 3.6. - -* :mod:`smtpd` has in the past always decoded the DATA portion of email - messages using the ``utf-8`` codec. This can now be controlled by the new - *decode_data* keyword to :class:`~smtpd.SMTPServer`. The default value is - ``True``, but this default is deprecated. Specify the *decode_data* keyword - with an appropriate value to avoid the deprecation warning. - -* Directly assigning values to the :attr:`~http.cookies.Morsel.key`, - :attr:`~http.cookies.Morsel.value` and - :attr:`~http.cookies.Morsel.coded_value` of :class:`~http.cookies.Morsel` - objects is deprecated. Use the :func:`~http.cookies.Morsel.set` method - instead. In addition, the undocumented *LegalChars* parameter of - :func:`~http.cookies.Morsel.set` is deprecated, and is now ignored. +The :mod:`formatter` module has now graduated to full deprecation and is still +slated for removal in Python 3.6. -* Passing a format string as keyword argument *format_string* to the - :meth:`~string.Formatter.format` method of the :class:`string.Formatter` - class has been deprecated. +The :func:`asyncio.async` function is deprecated in favor of +:func:`~asyncio.ensure_future`. -* :func:`platform.dist` and :func:`platform.linux_distribution` functions are - now deprecated and will be removed in Python 3.7. Linux distributions use - too many different ways of describing themselves, so the functionality is - left to a package. - (Contributed by Vajrasky Kok and Berker Peksag in :issue:`1322`.) +The :mod:`smtpd` module has in the past always decoded the DATA portion of +email messages using the ``utf-8`` codec. This can now be controlled by the +new *decode_data* keyword to :class:`~smtpd.SMTPServer`. The default value is +``True``, but this default is deprecated. Specify the *decode_data* keyword +with an appropriate value to avoid the deprecation warning. -* The previously undocumented ``from_function`` and ``from_builtin`` methods of - :class:`inspect.Signature` are deprecated. Use new - :meth:`inspect.Signature.from_callable` instead. (Contributed by Yury - Selivanov in :issue:`24248`.) +Directly assigning values to the :attr:`~http.cookies.Morsel.key`, +:attr:`~http.cookies.Morsel.value` and +:attr:`~http.cookies.Morsel.coded_value` of :class:`~http.cookies.Morsel` +objects is deprecated. Use the :func:`~http.cookies.Morsel.set` method +instead. In addition, the undocumented *LegalChars* parameter of +:func:`~http.cookies.Morsel.set` is deprecated, and is now ignored. -* :func:`inspect.getargspec` is deprecated and scheduled to be removed in - Python 3.6. (See :issue:`20438` for details.) +Passing a format string as keyword argument *format_string* to the +:meth:`~string.Formatter.format` method of the :class:`string.Formatter` +class has been deprecated. -* :func:`~inspect.getfullargspec`, :func:`~inspect.getargvalues`, - :func:`~inspect.getcallargs`, :func:`~inspect.getargvalues`, - :func:`~inspect.formatargspec`, and :func:`~inspect.formatargvalues` are - deprecated in favor of :func:`inspect.signature` API. (See :issue:`20438` - for details.) +The :func:`platform.dist` and :func:`platform.linux_distribution` functions +are now deprecated and will be removed in Python 3.7. Linux distributions use +too many different ways of describing themselves, so the functionality is +left to a package. +(Contributed by Vajrasky Kok and Berker Peksag in :issue:`1322`.) +The previously undocumented ``from_function`` and ``from_builtin`` methods of +:class:`inspect.Signature` are deprecated. Use new +:meth:`inspect.Signature.from_callable` instead. (Contributed by Yury +Selivanov in :issue:`24248`.) -Deprecated functions and types of the C API -------------------------------------------- +The :func:`inspect.getargspec` function is deprecated and scheduled to be +removed in Python 3.6. (See :issue:`20438` for details.) -* None yet. +The :mod:`inspect` :func:`~inspect.getfullargspec`, +:func:`~inspect.getargvalues`, :func:`~inspect.getcallargs`, +:func:`~inspect.getargvalues`, :func:`~inspect.formatargspec`, and +:func:`~inspect.formatargvalues` functions are deprecated in favor of +:func:`inspect.signature` API. +(Contributed by Yury Selivanov in :issue:`20438`.) - -Deprecated features -------------------- - -* None yet. +Use of ``re.LOCALE`` flag with str patterns or ``re.ASCII`` is now +deprecated. (Contributed by Serhiy Storchaka in :issue:`22407`.) Removed @@ -1110,7 +1854,8 @@ removed: * The concept of ``.pyo`` files has been removed. * The JoinableQueue class in the provisional asyncio module was deprecated - in 3.4.4 and is now removed (:issue:`23464`). + in 3.4.4 and is now removed. + (Contributed by A. Jesse Jiryu Davis in :issue:`23464`.) Porting to Python 3.5 @@ -1131,15 +1876,17 @@ Changes in the Python API error-prone and has been removed in Python 3.5. See :issue:`13936` for full details. -* :meth:`ssl.SSLSocket.send()` now raises either :exc:`ssl.SSLWantReadError` - or :exc:`ssl.SSLWantWriteError` on a non-blocking socket if the operation - would block. Previously, it would return 0. See :issue:`20951`. +* The :meth:`ssl.SSLSocket.send()` method now raises either + :exc:`ssl.SSLWantReadError` or :exc:`ssl.SSLWantWriteError` + on a non-blocking socket if the operation would block. Previously, + it would return ``0``. (Contributed by Nikolaus Rath in :issue:`20951`.) * The ``__name__`` attribute of generator is now set from the function name, instead of being set from the code name. Use ``gen.gi_code.co_name`` to retrieve the code name. Generators also have a new ``__qualname__`` attribute, the qualified name, which is now used for the representation - of a generator (``repr(gen)``). See :issue:`21205`. + of a generator (``repr(gen)``). + (Contributed by Victor Stinner in :issue:`21205`.) * The deprecated "strict" mode and argument of :class:`~html.parser.HTMLParser`, :meth:`HTMLParser.error`, and the :exc:`HTMLParserError` exception have been @@ -1150,8 +1897,8 @@ Changes in the Python API * Although it is not formally part of the API, it is worth noting for porting purposes (ie: fixing tests) that error messages that were previously of the form "'sometype' does not support the buffer protocol" are now of the form "a - bytes-like object is required, not 'sometype'". (Contributed by Ezio Melotti - in :issue:`16518`.) + :term:`bytes-like object` is required, not 'sometype'". + (Contributed by Ezio Melotti in :issue:`16518`.) * If the current directory is set to a directory that no longer exists then :exc:`FileNotFoundError` will no longer be raised and instead @@ -1170,14 +1917,14 @@ Changes in the Python API :exc:`DeprecationWarning` now, will be an error in Python 3.6). If the loader inherits from :class:`importlib.abc.Loader` then there is nothing to do, else simply define :meth:`~importlib.machinery.Loader.create_module` to return - ``None`` (:issue:`23014`). + ``None``. (Contributed by Brett Cannon in :issue:`23014`.) -* :func:`re.split` always ignored empty pattern matches, so the ``'x*'`` - pattern worked the same as ``'x+'``, and the ``'\b'`` pattern never worked. - Now :func:`re.split` raises a warning if the pattern could match +* The :func:`re.split` function always ignored empty pattern matches, so the + ``"x*"`` pattern worked the same as ``"x+"``, and the ``"\b"`` pattern never + worked. Now :func:`re.split` raises a warning if the pattern could match an empty string. For compatibility use patterns that never match an empty - string (e.g. ``'x+'`` instead of ``'x*'``). Patterns that could only match - an empty string (such as ``'\b'``) now raise an error. + string (e.g. ``"x+"`` instead of ``"x*"``). Patterns that could only match + an empty string (such as ``"\b"``) now raise an error. * The :class:`~http.cookies.Morsel` dict-like interface has been made self consistent: morsel comparison now takes the :attr:`~http.cookies.Morsel.key` @@ -1187,7 +1934,7 @@ Changes in the Python API :meth:`~http.cookies.Morsel.update` will now raise an exception if any of the keys in the update dictionary are invalid. In addition, the undocumented *LegalChars* parameter of :func:`~http.cookies.Morsel.set` is deprecated and - is now ignored. (:issue:`2211`) + is now ignored. (Contributed by Demian Brecht in :issue:`2211`.) * :pep:`488` has removed ``.pyo`` files from Python and introduced the optional ``opt-`` tag in ``.pyc`` file names. The @@ -1200,8 +1947,12 @@ Changes in the Python API in Python 3.5, all old `.pyo` files from previous versions of Python are invalid regardless of this PEP. -* The :mod:`socket` module now exports the CAN_RAW_FD_FRAMES constant on linux - 3.6 and greater. +* The :mod:`socket` module now exports the :data:`~socket.CAN_RAW_FD_FRAMES` + constant on linux 3.6 and greater. + +* The :func:`~ssl.cert_time_to_seconds` function now interprets the input time + as UTC and not as local time, per :rfc:`5280`. Additionally, the return + value is always an :class:`int`. (Contributed by Akira Li in :issue:`19940`.) * The ``pygettext.py`` Tool now uses the standard +NNNN format for timezones in the POT-Creation-Date header. @@ -1213,21 +1964,21 @@ Changes in the Python API * The :meth:`str.startswith` and :meth:`str.endswith` methods no longer return ``True`` when finding the empty string and the indexes are completely out of - range. See :issue:`24284`. + range. (Contributed by Serhiy Storchaka in :issue:`24284`.) * The :func:`inspect.getdoc` function now returns documentation strings inherited from base classes. Documentation strings no longer need to be duplicated if the inherited documentation is appropriate. To suppress an inherited string, an empty string must be specified (or the documentation may be filled in). This change affects the output of the :mod:`pydoc` - module and the :func:`help` function. See :issue:`15582`. + module and the :func:`help` function. + (Contributed by Serhiy Storchaka in :issue:`15582`.) Changes in the C API -------------------- * The undocumented :c:member:`~PyMemoryViewObject.format` member of the (non-public) :c:type:`PyMemoryViewObject` structure has been removed. - All extensions relying on the relevant parts in ``memoryobject.h`` must be rebuilt. @@ -1237,13 +1988,15 @@ Changes in the C API * Removed non-documented macro :c:macro:`PyObject_REPR` which leaked references. Use format character ``%R`` in :c:func:`PyUnicode_FromFormat`-like functions to format the :func:`repr` of the object. + (Contributed by Serhiy Storchaka in :issue:`22453`.) * Because the lack of the :attr:`__module__` attribute breaks pickling and introspection, a deprecation warning now is raised for builtin type without the :attr:`__module__` attribute. Would be an AttributeError in future. - (:issue:`20204`) + (Contributed by Serhiy Storchaka in :issue:`20204`.) * As part of :pep:`492` implementation, ``tp_reserved`` slot of :c:type:`PyTypeObject` was replaced with a :c:member:`tp_as_async` slot. Refer to :ref:`coro-objects` for new types, structures and functions. + diff --git a/Include/object.h b/Include/object.h index 8afcbe9..4d286ef 100644 --- a/Include/object.h +++ b/Include/object.h @@ -351,7 +351,8 @@ typedef struct _typeobject { printfunc tp_print; getattrfunc tp_getattr; setattrfunc tp_setattr; - PyAsyncMethods *tp_as_async; /* formerly known as tp_compare or tp_reserved */ + PyAsyncMethods *tp_as_async; /* formerly known as tp_compare (Python 2) + or tp_reserved (Python 3) */ reprfunc tp_repr; /* Method suites for standard classes */ diff --git a/Lib/_pyio.py b/Lib/_pyio.py index 50ad9ff..f47df91 100644 --- a/Lib/_pyio.py +++ b/Lib/_pyio.py @@ -8,12 +8,13 @@ import codecs import errno import array import stat +import sys # Import _thread instead of threading to reduce startup cost try: from _thread import allocate_lock as Lock except ImportError: from _dummy_thread import allocate_lock as Lock -if os.name == 'win32': +if sys.platform in {'win32', 'cygwin'}: from msvcrt import setmode as _setmode else: _setmode = None @@ -720,6 +720,11 @@ class FieldStorage: self.bytes_read += len(hdr_text) parser.feed(hdr_text.decode(self.encoding, self.errors)) headers = parser.close() + + # Some clients add Content-Length for part headers, ignore them + if 'content-length' in headers: + del headers['content-length'] + part = klass(self.fp, headers, ib, environ, keep_blank_values, strict_parsing,self.limit-self.bytes_read, self.encoding, self.errors) diff --git a/Lib/collections/__init__.py b/Lib/collections/__init__.py index 80dc4f6..943cbca 100644 --- a/Lib/collections/__init__.py +++ b/Lib/collections/__init__.py @@ -320,23 +320,14 @@ class {typename}(tuple): 'Return a nicely formatted representation string' return self.__class__.__name__ + '({repr_fmt})' % self - @property - def __dict__(self): - 'A new OrderedDict mapping field names to their values' - return OrderedDict(zip(self._fields, self)) - def _asdict(self): 'Return a new OrderedDict which maps field names to their values.' - return self.__dict__ + return OrderedDict(zip(self._fields, self)) def __getnewargs__(self): 'Return self as a plain tuple. Used by copy and pickle.' return tuple(self) - def __getstate__(self): - 'Exclude the OrderedDict from pickling' - return None - {field_defs} """ @@ -901,9 +892,8 @@ class ChainMap(MutableMapping): __copy__ = copy def new_child(self, m=None): # like Django's Context.push() - ''' - New ChainMap with a new map followed by all previous maps. If no - map is provided, an empty dict is used. + '''New ChainMap with a new map followed by all previous maps. + If no map is provided, an empty dict is used. ''' if m is None: m = {} diff --git a/Lib/configparser.py b/Lib/configparser.py index ecd0660..3a9fb56 100644 --- a/Lib/configparser.py +++ b/Lib/configparser.py @@ -263,12 +263,9 @@ class InterpolationMissingOptionError(InterpolationError): """A string substitution required a setting which was not available.""" def __init__(self, option, section, rawval, reference): - msg = ("Bad value substitution:\n" - "\tsection: [%s]\n" - "\toption : %s\n" - "\tkey : %s\n" - "\trawval : %s\n" - % (section, option, reference, rawval)) + msg = ("Bad value substitution: option {!r} in section {!r} contains " + "an interpolation key {!r} which is not a valid option name. " + "Raw value: {!r}".format(option, section, reference, rawval)) InterpolationError.__init__(self, option, section, msg) self.reference = reference self.args = (option, section, rawval, reference) @@ -286,11 +283,11 @@ class InterpolationDepthError(InterpolationError): """Raised when substitutions are nested too deeply.""" def __init__(self, option, section, rawval): - msg = ("Value interpolation too deeply recursive:\n" - "\tsection: [%s]\n" - "\toption : %s\n" - "\trawval : %s\n" - % (section, option, rawval)) + msg = ("Recursion limit exceeded in value substitution: option {!r} " + "in section {!r} contains an interpolation key which " + "cannot be substituted in {} steps. Raw value: {!r}" + "".format(option, section, MAX_INTERPOLATION_DEPTH, + rawval)) InterpolationError.__init__(self, option, section, msg) self.args = (option, section, rawval) @@ -406,8 +403,9 @@ class BasicInterpolation(Interpolation): def _interpolate_some(self, parser, option, accum, rest, section, map, depth): + rawval = parser.get(section, option, raw=True, fallback=rest) if depth > MAX_INTERPOLATION_DEPTH: - raise InterpolationDepthError(option, section, rest) + raise InterpolationDepthError(option, section, rawval) while rest: p = rest.find("%") if p < 0: @@ -432,7 +430,7 @@ class BasicInterpolation(Interpolation): v = map[var] except KeyError: raise InterpolationMissingOptionError( - option, section, rest, var) from None + option, section, rawval, var) from None if "%" in v: self._interpolate_some(parser, option, accum, v, section, map, depth + 1) @@ -466,8 +464,9 @@ class ExtendedInterpolation(Interpolation): def _interpolate_some(self, parser, option, accum, rest, section, map, depth): + rawval = parser.get(section, option, raw=True, fallback=rest) if depth > MAX_INTERPOLATION_DEPTH: - raise InterpolationDepthError(option, section, rest) + raise InterpolationDepthError(option, section, rawval) while rest: p = rest.find("$") if p < 0: @@ -504,7 +503,7 @@ class ExtendedInterpolation(Interpolation): "More than one ':' found: %r" % (rest,)) except (KeyError, NoSectionError, NoOptionError): raise InterpolationMissingOptionError( - option, section, rest, ":".join(path)) from None + option, section, rawval, ":".join(path)) from None if "$" in v: self._interpolate_some(parser, opt, accum, v, sect, dict(parser.items(sect, raw=True)), diff --git a/Lib/distutils/_msvccompiler.py b/Lib/distutils/_msvccompiler.py index 82b78a0..03a5f10 100644 --- a/Lib/distutils/_msvccompiler.py +++ b/Lib/distutils/_msvccompiler.py @@ -100,7 +100,7 @@ def _get_vc_env(plat_spec): (line.partition('=') for line in out.splitlines()) if key and value } - + if vcruntime: env['py_vcruntime_redist'] = vcruntime return env @@ -236,7 +236,7 @@ class MSVCCompiler(CCompiler) : '/nologo', '/Ox', '/W3', '/GL', '/DNDEBUG' ] self.compile_options.append('/MD' if self._vcruntime_redist else '/MT') - + self.compile_options_debug = [ '/nologo', '/Od', '/MDd', '/Zi', '/W3', '/D_DEBUG' ] diff --git a/Lib/distutils/tests/test_msvccompiler.py b/Lib/distutils/tests/test_msvccompiler.py index 874d603..c4d911f 100644 --- a/Lib/distutils/tests/test_msvccompiler.py +++ b/Lib/distutils/tests/test_msvccompiler.py @@ -77,7 +77,7 @@ class msvccompilerTestCase(support.TempdirManager, compiler.initialize() dll = compiler._vcruntime_redist self.assertTrue(os.path.isfile(dll)) - + compiler._copy_vcruntime(tempdir) self.assertFalse(os.path.isfile(os.path.join( diff --git a/Lib/functools.py b/Lib/functools.py index 09df068..06a4ff1 100644 --- a/Lib/functools.py +++ b/Lib/functools.py @@ -567,7 +567,7 @@ def _c3_merge(sequences): break # reject the current head, it appears later else: break - if not candidate: + if candidate is None: raise RuntimeError("Inconsistent hierarchy") result.append(candidate) # remove the chosen candidate diff --git a/Lib/html/parser.py b/Lib/html/parser.py index 390d4cc..43e6411 100644 --- a/Lib/html/parser.py +++ b/Lib/html/parser.py @@ -139,7 +139,15 @@ class HTMLParser(_markupbase.ParserBase): if self.convert_charrefs and not self.cdata_elem: j = rawdata.find('<', i) if j < 0: - if not end: + # if we can't find the next <, either we are at the end + # or there's more text incoming. If the latter is True, + # we can't pass the text to handle_data in case we have + # a charref cut in half at end. Try to determine if + # this is the case before proceding by looking for an + # & near the end and see if it's followed by a space or ;. + amppos = rawdata.rfind('&', max(i, n-34)) + if (amppos >= 0 and + not re.compile(r'[\s;]').search(rawdata, amppos)): break # wait till we get all the text j = n else: diff --git a/Lib/http/server.py b/Lib/http/server.py index fd13be3..6a0e6fe 100644 --- a/Lib/http/server.py +++ b/Lib/http/server.py @@ -1167,8 +1167,7 @@ def test(HandlerClass=BaseHTTPRequestHandler, ServerClass=HTTPServer, protocol="HTTP/1.0", port=8000, bind=""): """Test the HTTP request handler class. - This runs an HTTP server on port 8000 (or the first command line - argument). + This runs an HTTP server on port 8000 (or the port argument). """ server_address = (bind, port) diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt index 2d8ce54..be063c4 100644 --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -2,6 +2,19 @@ What's New in IDLE 3.5.0? ========================= *Release date: 2015-09-13* ?? +- Issue #23672: Allow Idle to edit and run files with astral chars in name. + Patch by Mohd Sanad Zaki Rizvi. + +- Issue 24745: Idle editor default font. Switch from Courier to + platform-sensitive TkFixedFont. This should not affect current customized + font selections. If there is a problem, edit $HOME/.idlerc/config-main.cfg + and remove 'fontxxx' entries from [Editor Window]. Patch by Mark Roseman. + +- Issue #21192: Idle editor. When a file is run, put its name in the restart bar. + Do not print false prompts. Original patch by Adnan Umer. + +- Issue #13884: Idle menus. Remove tearoff lines. Patch by Roger Serwy. + - Issue #23184: remove unused names and imports in idlelib. Initial patch by Al Sweigart. diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py index 5854cf9..90fc689 100755 --- a/Lib/idlelib/PyShell.py +++ b/Lib/idlelib/PyShell.py @@ -1043,6 +1043,7 @@ class PyShell(OutputWindow): self.write("Python %s on %s\n%s\n%s" % (sys.version, sys.platform, self.COPYRIGHT, nosub)) + self.text.focus_force() self.showprompt() import tkinter tkinter._default_root = None # 03Jan04 KBK What's this? diff --git a/Lib/idlelib/ScriptBinding.py b/Lib/idlelib/ScriptBinding.py index e8cb2fc..e5636df 100644 --- a/Lib/idlelib/ScriptBinding.py +++ b/Lib/idlelib/ScriptBinding.py @@ -69,7 +69,7 @@ class ScriptBinding: try: tabnanny.process_tokens(tokenize.generate_tokens(f.readline)) except tokenize.TokenError as msg: - msgtxt, (lineno, start) = msg + msgtxt, (lineno, start) = msg.args self.editwin.gotoline(lineno) self.errorbox("Tabnanny Tokenizing Error", "Token Error: %s" % msgtxt) diff --git a/Lib/idlelib/StackViewer.py b/Lib/idlelib/StackViewer.py index b1e5e26..ccc755c 100644 --- a/Lib/idlelib/StackViewer.py +++ b/Lib/idlelib/StackViewer.py @@ -10,8 +10,7 @@ from idlelib.PyShell import PyShellFileList def StackBrowser(root, flist=None, tb=None, top=None): if top is None: - from tkinter import Toplevel - top = Toplevel(root) + top = tk.Toplevel(root) sc = ScrolledCanvas(top, bg="white", highlightthickness=0) sc.frame.pack(expand=1, fill="both") item = StackTreeItem(flist, tb) @@ -108,12 +107,9 @@ class VariablesTreeItem(ObjectTreeItem): def IsExpandable(self): return len(self.object) > 0 - def keys(self): - return list(self.object.keys()) - def GetSubList(self): sublist = [] - for key in self.keys(): + for key in self.object.keys(): try: value = self.object[key] except KeyError: @@ -124,6 +120,9 @@ class VariablesTreeItem(ObjectTreeItem): sublist.append(item) return sublist + def keys(self): # unused, left for possible 3rd party use + return list(self.object.keys()) + def _stack_viewer(parent): root = tk.Tk() root.title("Test StackViewer") diff --git a/Lib/idlelib/configDialog.py b/Lib/idlelib/configDialog.py index 9ed6336..b70cb60 100644 --- a/Lib/idlelib/configDialog.py +++ b/Lib/idlelib/configDialog.py @@ -1201,9 +1201,6 @@ class VerticalScrolledFrame(Frame): # update the scrollbars to match the size of the inner frame size = (interior.winfo_reqwidth(), interior.winfo_reqheight()) canvas.config(scrollregion="0 0 %s %s" % size) - if interior.winfo_reqwidth() != canvas.winfo_width(): - # update the canvas's width to fit the inner frame - canvas.config(width=interior.winfo_reqwidth()) interior.bind('<Configure>', _configure_interior) def _configure_canvas(event): @@ -1323,38 +1320,56 @@ class ConfigExtensionsDialog(Toplevel): def create_widgets(self): """Create the dialog's widgets.""" + self.extension_names = StringVar(self) self.rowconfigure(0, weight=1) - self.rowconfigure(1, weight=0) - self.columnconfigure(0, weight=1) - - # create the tabbed pages - self.tabbed_page_set = TabbedPageSet( - self, page_names=self.extensions.keys(), - n_rows=None, max_tabs_per_row=5, - page_class=TabbedPageSet.PageRemove) - self.tabbed_page_set.grid(row=0, column=0, sticky=NSEW) - for ext_name in self.extensions: - self.create_tab_page(ext_name) - - self.create_action_buttons().grid(row=1) + self.columnconfigure(2, weight=1) + self.extension_list = Listbox(self, listvariable=self.extension_names, + selectmode='browse') + self.extension_list.bind('<<ListboxSelect>>', self.extension_selected) + scroll = Scrollbar(self, command=self.extension_list.yview) + self.extension_list.yscrollcommand=scroll.set + self.details_frame = LabelFrame(self, width=250, height=250) + self.extension_list.grid(column=0, row=0, sticky='nws') + scroll.grid(column=1, row=0, sticky='ns') + self.details_frame.grid(column=2, row=0, sticky='nsew', padx=[10, 0]) + self.configure(padx=10, pady=10) + self.config_frame = {} + self.current_extension = None + + self.outerframe = self # TEMPORARY + self.tabbed_page_set = self.extension_list # TEMPORARY + + # create the individual pages + ext_names = '' + for ext_name in sorted(self.extensions): + self.create_extension_frame(ext_name) + ext_names = ext_names + '{' + ext_name + '} ' + self.extension_names.set(ext_names) + self.extension_list.selection_set(0) + self.extension_selected(None) + self.create_action_buttons().grid(row=1, columnspan=3) + + def extension_selected(self, event): + newsel = self.extension_list.curselection() + if newsel: + newsel = self.extension_list.get(newsel) + if newsel is None or newsel != self.current_extension: + if self.current_extension: + self.details_frame.config(text='') + self.config_frame[self.current_extension].grid_forget() + self.current_extension = None + if newsel: + self.details_frame.config(text=newsel) + self.config_frame[newsel].grid(column=0, row=0, sticky='nsew') + self.current_extension = newsel create_action_buttons = ConfigDialog.create_action_buttons - def create_tab_page(self, ext_name): - """Create the page for an extension.""" - - page = LabelFrame(self.tabbed_page_set.pages[ext_name].frame, - border=2, padx=2, relief=GROOVE, - text=' %s ' % ext_name) - page.pack(fill=BOTH, expand=True, padx=12, pady=2) - - # create the scrollable frame which will contain the entries - scrolled_frame = VerticalScrolledFrame(page, pady=2, height=250) - scrolled_frame.pack(side=BOTTOM, fill=BOTH, expand=TRUE) - entry_area = scrolled_frame.interior - entry_area.columnconfigure(0, weight=0) - entry_area.columnconfigure(1, weight=1) - + def create_extension_frame(self, ext_name): + """Create a frame holding the widgets to configure one extension""" + f = VerticalScrolledFrame(self.details_frame, height=250, width=250) + self.config_frame[ext_name] = f + entry_area = f.interior # create an entry for each configuration option for row, opt in enumerate(self.extensions[ext_name]): # create a row with a label and entry/checkbutton @@ -1365,15 +1380,15 @@ class ConfigExtensionsDialog(Toplevel): Checkbutton(entry_area, textvariable=var, variable=var, onvalue='True', offvalue='False', indicatoron=FALSE, selectcolor='', width=8 - ).grid(row=row, column=1, sticky=W, padx=7) + ).grid(row=row, column=1, sticky=W, padx=7) elif opt['type'] == 'int': Entry(entry_area, textvariable=var, validate='key', - validatecommand=(self.is_int, '%P') - ).grid(row=row, column=1, sticky=NSEW, padx=7) + validatecommand=(self.is_int, '%P') + ).grid(row=row, column=1, sticky=NSEW, padx=7) else: Entry(entry_area, textvariable=var - ).grid(row=row, column=1, sticky=NSEW, padx=7) + ).grid(row=row, column=1, sticky=NSEW, padx=7) return diff --git a/Lib/idlelib/idle_test/test_warning.py b/Lib/idlelib/idle_test/test_warning.py index 18627dd..54ac993 100644 --- a/Lib/idlelib/idle_test/test_warning.py +++ b/Lib/idlelib/idle_test/test_warning.py @@ -68,6 +68,15 @@ class ShellWarnTest(unittest.TestCase): 'Test', UserWarning, 'test_warning.py', 99, f, 'Line of code') self.assertEqual(shellmsg.splitlines(), f.getvalue().splitlines()) +class ImportWarnTest(unittest.TestCase): + def test_idlever(self): + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter("always") + import idlelib.idlever + self.assertEqual(len(w), 1) + self.assertTrue(issubclass(w[-1].category, DeprecationWarning)) + self.assertIn("version", str(w[-1].message)) + if __name__ == '__main__': unittest.main(verbosity=2, exit=False) diff --git a/Lib/idlelib/idlever.py b/Lib/idlelib/idlever.py index 563d933..13c68b8 100644 --- a/Lib/idlelib/idlever.py +++ b/Lib/idlelib/idlever.py @@ -1,4 +1,12 @@ -"""Unused by Idle: there is no separate Idle version anymore. -Kept only for possible existing extension use.""" +""" +The separate Idle version was eliminated years ago; +idlelib.idlever is no longer used by Idle +and will be removed in 3.6 or later. Use + from sys import version + IDLE_VERSION = version[:version.index(' ')] +""" +# Kept for now only for possible existing extension use +import warnings as w +w.warn(__doc__, DeprecationWarning) from sys import version IDLE_VERSION = version[:version.index(' ')] @@ -1669,6 +1669,9 @@ def main(): # In most cases SystemExit does not warrant a post-mortem session. print("The program exited via sys.exit(). Exit status:", end=' ') print(sys.exc_info()[1]) + except SyntaxError: + traceback.print_exc() + sys.exit(1) except: traceback.print_exc() print("Uncaught exception. Entering post mortem debugging") diff --git a/Lib/shutil.py b/Lib/shutil.py index a5da587..3f4b6bf 100644 --- a/Lib/shutil.py +++ b/Lib/shutil.py @@ -679,7 +679,16 @@ def _make_zipfile(base_name, base_dir, verbose=0, dry_run=0, logger=None): if not dry_run: with zipfile.ZipFile(zip_filename, "w", compression=zipfile.ZIP_DEFLATED) as zf: + path = os.path.normpath(base_dir) + zf.write(path, path) + if logger is not None: + logger.info("adding '%s'", path) for dirpath, dirnames, filenames in os.walk(base_dir): + for name in sorted(dirnames): + path = os.path.normpath(os.path.join(dirpath, name)) + zf.write(path, path) + if logger is not None: + logger.info("adding '%s'", path) for name in filenames: path = os.path.normpath(os.path.join(dirpath, name)) if os.path.isfile(path): diff --git a/Lib/test/test_binascii.py b/Lib/test/test_binascii.py index 4e67887..8367afe 100644 --- a/Lib/test/test_binascii.py +++ b/Lib/test/test_binascii.py @@ -178,6 +178,8 @@ class BinASCIITest(unittest.TestCase): self.assertEqual(binascii.unhexlify(self.type2test(t)), u) def test_qp(self): + binascii.a2b_qp(data=b"", header=False) # Keyword arguments allowed + # A test for SF bug 534347 (segfaults without the proper fix) try: binascii.a2b_qp(b"", **{1:1}) @@ -185,6 +187,7 @@ class BinASCIITest(unittest.TestCase): pass else: self.fail("binascii.a2b_qp(**{1:1}) didn't raise TypeError") + self.assertEqual(binascii.a2b_qp(b"= "), b"= ") self.assertEqual(binascii.a2b_qp(b"=="), b"=") self.assertEqual(binascii.a2b_qp(b"=AX"), b"=AX") diff --git a/Lib/test/test_cgi.py b/Lib/test/test_cgi.py index a7a9d02..ab9f6ab 100644 --- a/Lib/test/test_cgi.py +++ b/Lib/test/test_cgi.py @@ -326,6 +326,24 @@ Content-Type: text/plain got = getattr(files[x], k) self.assertEqual(got, exp) + def test_fieldstorage_part_content_length(self): + BOUNDARY = "JfISa01" + POSTDATA = """--JfISa01 +Content-Disposition: form-data; name="submit-name" +Content-Length: 5 + +Larry +--JfISa01""" + env = { + 'REQUEST_METHOD': 'POST', + 'CONTENT_TYPE': 'multipart/form-data; boundary={}'.format(BOUNDARY), + 'CONTENT_LENGTH': str(len(POSTDATA))} + fp = BytesIO(POSTDATA.encode('latin-1')) + fs = cgi.FieldStorage(fp, environ=env, encoding="latin-1") + self.assertEqual(len(fs.list), 1) + self.assertEqual(fs.list[0].name, 'submit-name') + self.assertEqual(fs.list[0].value, 'Larry') + def test_fieldstorage_as_context_manager(self): fp = BytesIO(b'x' * 10) env = {'REQUEST_METHOD': 'PUT'} diff --git a/Lib/test/test_collections.py b/Lib/test/test_collections.py index c2d03ee..4124f91 100644 --- a/Lib/test/test_collections.py +++ b/Lib/test/test_collections.py @@ -257,7 +257,6 @@ class TestNamedTuple(unittest.TestCase): self.assertEqual(p._fields, ('x', 'y')) # test _fields attribute self.assertEqual(p._replace(x=1), (1, 22)) # test _replace method self.assertEqual(p._asdict(), dict(x=11, y=22)) # test _asdict method - self.assertEqual(vars(p), p._asdict()) # verify that vars() works try: p._replace(x=1, error=2) @@ -412,6 +411,17 @@ class TestNamedTuple(unittest.TestCase): globals().pop('NTColor', None) # clean-up after this test + def test_namedtuple_subclass_issue_24931(self): + class Point(namedtuple('_Point', ['x', 'y'])): + pass + + a = Point(3, 4) + self.assertEqual(a._asdict(), OrderedDict([('x', 3), ('y', 4)])) + + a.w = 5 + self.assertEqual(a.__dict__, {'w': 5}) + + ################################################################################ ### Abstract Base Classes ################################################################################ diff --git a/Lib/test/test_configparser.py b/Lib/test/test_configparser.py index 470d2cd..71a8f3f 100644 --- a/Lib/test/test_configparser.py +++ b/Lib/test/test_configparser.py @@ -847,7 +847,8 @@ class ConfigParserTestCase(BasicTestCase, unittest.TestCase): "something with lots of interpolation (10 steps)") e = self.get_error(cf, configparser.InterpolationDepthError, "Foo", "bar11") if self.interpolation == configparser._UNSET: - self.assertEqual(e.args, ("bar11", "Foo", "%(with1)s")) + self.assertEqual(e.args, ("bar11", "Foo", + "something %(with11)s lots of interpolation (11 steps)")) elif isinstance(self.interpolation, configparser.LegacyInterpolation): self.assertEqual(e.args, ("bar11", "Foo", "something %(with11)s lots of interpolation (11 steps)")) @@ -861,7 +862,7 @@ class ConfigParserTestCase(BasicTestCase, unittest.TestCase): self.assertEqual(e.option, "name") if self.interpolation == configparser._UNSET: self.assertEqual(e.args, ('name', 'Interpolation Error', - '', 'reference')) + '%(reference)s', 'reference')) elif isinstance(self.interpolation, configparser.LegacyInterpolation): self.assertEqual(e.args, ('name', 'Interpolation Error', '%(reference)s', 'reference')) @@ -1177,7 +1178,7 @@ class ConfigParserTestCaseExtendedInterpolation(BasicTestCase, unittest.TestCase with self.assertRaises(exception_class) as cm: cf['interpolated']['$trying'] self.assertEqual(cm.exception.reference, 'dollars:${sick') - self.assertEqual(cm.exception.args[2], '}') #rawval + self.assertEqual(cm.exception.args[2], '${dollars:${sick}}') #rawval def test_case_sensitivity_basic(self): ini = textwrap.dedent(""" diff --git a/Lib/test/test_functools.py b/Lib/test/test_functools.py index ac211c4..ae929ec 100644 --- a/Lib/test/test_functools.py +++ b/Lib/test/test_functools.py @@ -1491,6 +1491,24 @@ class TestSingleDispatch(unittest.TestCase): many_abcs = [c.Mapping, c.Sized, c.Callable, c.Container, c.Iterable] self.assertEqual(mro(X, abcs=many_abcs), expected) + def test_false_meta(self): + # see issue23572 + class MetaA(type): + def __len__(self): + return 0 + class A(metaclass=MetaA): + pass + class AA(A): + pass + @functools.singledispatch + def fun(a): + return 'base A' + @fun.register(A) + def _(a): + return 'fun A' + aa = AA() + self.assertEqual(fun(aa), 'fun A') + def test_mro_conflicts(self): c = collections @functools.singledispatch diff --git a/Lib/test/test_gdb.py b/Lib/test/test_gdb.py index 0322677..db777be 100644 --- a/Lib/test/test_gdb.py +++ b/Lib/test/test_gdb.py @@ -21,19 +21,34 @@ except ImportError: from test import support from test.support import run_unittest, findfile, python_is_optimized -try: - gdb_version, _ = subprocess.Popen(["gdb", "-nx", "--version"], - stdout=subprocess.PIPE).communicate() -except OSError: - # This is what "no gdb" looks like. There may, however, be other - # errors that manifest this way too. - raise unittest.SkipTest("Couldn't find gdb on the path") -gdb_version_number = re.search(b"^GNU gdb [^\d]*(\d+)\.(\d)", gdb_version) -gdb_major_version = int(gdb_version_number.group(1)) -gdb_minor_version = int(gdb_version_number.group(2)) +def get_gdb_version(): + try: + proc = subprocess.Popen(["gdb", "-nx", "--version"], + stdout=subprocess.PIPE, + universal_newlines=True) + with proc: + version = proc.communicate()[0] + except OSError: + # This is what "no gdb" looks like. There may, however, be other + # errors that manifest this way too. + raise unittest.SkipTest("Couldn't find gdb on the path") + + # Regex to parse: + # 'GNU gdb (GDB; SUSE Linux Enterprise 12) 7.7\n' -> 7.7 + # 'GNU gdb (GDB) Fedora 7.9.1-17.fc22\n' -> 7.9 + # 'GNU gdb 6.1.1 [FreeBSD]\n' -> 6.1 + # 'GNU gdb (GDB) Fedora (7.5.1-37.fc18)\n' -> 7.5 + match = re.search(r"^GNU gdb.*?\b(\d+)\.(\d)", version) + if match is None: + raise Exception("unable to parse GDB version: %r" % version) + return (version, int(match.group(1)), int(match.group(2))) + +gdb_version, gdb_major_version, gdb_minor_version = get_gdb_version() if gdb_major_version < 7: - raise unittest.SkipTest("gdb versions before 7.0 didn't support python embedding" - " Saw:\n" + gdb_version.decode('ascii', 'replace')) + raise unittest.SkipTest("gdb versions before 7.0 didn't support python " + "embedding. Saw %s.%s:\n%s" + % (gdb_major_version, gdb_minor_version, + gdb_version)) if not sysconfig.is_python_build(): raise unittest.SkipTest("test_gdb only works on source builds at the moment.") @@ -59,9 +74,12 @@ def run_gdb(*args, **env_vars): base_cmd = ('gdb', '--batch', '-nx') if (gdb_major_version, gdb_minor_version) >= (7, 4): base_cmd += ('-iex', 'add-auto-load-safe-path ' + checkout_hook_path) - out, err = subprocess.Popen(base_cmd + args, - stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env, - ).communicate() + proc = subprocess.Popen(base_cmd + args, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + env=env) + with proc: + out, err = proc.communicate() return out.decode('utf-8', 'replace'), err.decode('utf-8', 'replace') # Verify that "gdb" was built with the embedded python support enabled: @@ -880,8 +898,8 @@ class PyLocalsTests(DebuggerTests): def test_main(): if support.verbose: - print("GDB version:") - for line in os.fsdecode(gdb_version).splitlines(): + print("GDB version %s.%s:" % (gdb_major_version, gdb_minor_version)) + for line in gdb_version.splitlines(): print(" " * 4 + line) run_unittest(PrettyPrintTests, PyListTests, diff --git a/Lib/test/test_glob.py b/Lib/test/test_glob.py index 21b0153..926588e 100644 --- a/Lib/test/test_glob.py +++ b/Lib/test/test_glob.py @@ -242,9 +242,7 @@ class GlobTests(unittest.TestCase): ('a', 'bcd', 'EF'), ('a', 'bcd', 'efg'))) eq(self.rglob('a', '**', 'bcd'), self.joins(('a', 'bcd'))) - predir = os.path.abspath(os.curdir) - try: - os.chdir(self.tempdir) + with change_cwd(self.tempdir): join = os.path.join eq(glob.glob('**', recursive=True), [join(*i) for i in full]) eq(glob.glob(join('**', ''), recursive=True), @@ -256,8 +254,6 @@ class GlobTests(unittest.TestCase): if can_symlink(): expect += [join('sym3', 'EF')] eq(glob.glob(join('**', 'EF'), recursive=True), expect) - finally: - os.chdir(predir) @skip_unless_symlink diff --git a/Lib/test/test_htmlparser.py b/Lib/test/test_htmlparser.py index de8f3e8..11420b2c 100644 --- a/Lib/test/test_htmlparser.py +++ b/Lib/test/test_htmlparser.py @@ -72,9 +72,6 @@ class EventCollectorExtra(EventCollector): class EventCollectorCharrefs(EventCollector): - def get_events(self): - return self.events - def handle_charref(self, data): self.fail('This should never be called with convert_charrefs=True') @@ -633,6 +630,18 @@ text ] self._run_check(html, expected) + def test_convert_charrefs_dropped_text(self): + # #23144: make sure that all the events are triggered when + # convert_charrefs is True, even if we don't call .close() + parser = EventCollector(convert_charrefs=True) + # before the fix, bar & baz was missing + parser.feed("foo <a>link</a> bar & baz") + self.assertEqual( + parser.get_events(), + [('data', 'foo '), ('starttag', 'a', []), ('data', 'link'), + ('endtag', 'a'), ('data', ' bar & baz')] + ) + class AttributesTestCase(TestCaseBase): diff --git a/Lib/test/test_mmap.py b/Lib/test/test_mmap.py index 3de84e8..0f25742 100644 --- a/Lib/test/test_mmap.py +++ b/Lib/test/test_mmap.py @@ -731,7 +731,10 @@ class LargeMmapTests(unittest.TestCase): f.write(tail) f.flush() except (OSError, OverflowError): - f.close() + try: + f.close() + except (OSError, OverflowError): + pass raise unittest.SkipTest("filesystem does not have largefile support") return f diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py index d91f58c..0b77835 100644 --- a/Lib/test/test_os.py +++ b/Lib/test/test_os.py @@ -85,7 +85,7 @@ HAVE_WHEEL_GROUP = sys.platform.startswith('freebsd') and os.getgid() == 0 # Tests creating TESTFN class FileTests(unittest.TestCase): def setUp(self): - if os.path.exists(support.TESTFN): + if os.path.lexists(support.TESTFN): os.unlink(support.TESTFN) tearDown = setUp @@ -209,6 +209,19 @@ class FileTests(unittest.TestCase): with open(TESTFN2, 'r') as f: self.assertEqual(f.read(), "1") + def test_open_keywords(self): + f = os.open(path=__file__, flags=os.O_RDONLY, mode=0o777, + dir_fd=None) + os.close(f) + + def test_symlink_keywords(self): + symlink = support.get_attribute(os, "symlink") + try: + symlink(src='target', dst=support.TESTFN, + target_is_directory=False, dir_fd=None) + except (NotImplementedError, OSError): + pass # No OS support or unprivileged user + # Test attributes on return values from os.*stat* family. class StatAttributeTests(unittest.TestCase): @@ -2313,6 +2326,14 @@ class TestSendfile(unittest.TestCase): os.sendfile(self.sockno, self.fileno, -1, 4096) self.assertEqual(cm.exception.errno, errno.EINVAL) + def test_keywords(self): + # Keyword arguments should be supported + os.sendfile(out=self.sockno, offset=0, count=4096, + **{'in': self.fileno}) + if self.SUPPORT_HEADERS_TRAILERS: + os.sendfile(self.sockno, self.fileno, offset=0, count=4096, + headers=(), trailers=(), flags=0) + # --- headers / trailers tests @requires_headers_trailers diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py index ec8346c..35044ad 100644 --- a/Lib/test/test_pdb.py +++ b/Lib/test/test_pdb.py @@ -1043,6 +1043,18 @@ class PdbTestCase(unittest.TestCase): self.assertNotIn('Error', stdout.decode(), "Got an error running test script under PDB") + def test_issue16180(self): + # A syntax error in the debuggee. + script = "def f: pass\n" + commands = '' + expected = "SyntaxError:" + stdout, stderr = self.run_pdb(script, commands) + self.assertIn(expected, stdout, + '\n\nExpected:\n{}\nGot:\n{}\n' + 'Fail to handle a syntax error in the debuggee.' + .format(expected, stdout)) + + def tearDown(self): support.unlink(support.TESTFN) diff --git a/Lib/test/test_pep277.py b/Lib/test/test_pep277.py index 6c833a8..98c716b 100644 --- a/Lib/test/test_pep277.py +++ b/Lib/test/test_pep277.py @@ -158,17 +158,11 @@ class UnicodeFileTests(unittest.TestCase): def test_directory(self): dirname = os.path.join(support.TESTFN, 'Gr\xfc\xdf-\u66e8\u66e9\u66eb') filename = '\xdf-\u66e8\u66e9\u66eb' - oldwd = os.getcwd() - os.mkdir(dirname) - os.chdir(dirname) - try: + with support.temp_cwd(dirname): with open(filename, 'wb') as f: f.write((filename + '\n').encode("utf-8")) os.access(filename,os.R_OK) os.remove(filename) - finally: - os.chdir(oldwd) - os.rmdir(dirname) class UnicodeNFCFileTests(UnicodeFileTests): diff --git a/Lib/test/test_popen.py b/Lib/test/test_popen.py index 8958db0..da01a87 100644 --- a/Lib/test/test_popen.py +++ b/Lib/test/test_popen.py @@ -57,5 +57,9 @@ class PopenTest(unittest.TestCase): with os.popen("echo hello") as f: self.assertEqual(list(f), ["hello\n"]) + def test_keywords(self): + with os.popen(cmd="exit 0", mode="w", buffering=-1): + pass + if __name__ == "__main__": unittest.main() diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py index 77e5b0c4..2a59c38 100644 --- a/Lib/test/test_posix.py +++ b/Lib/test/test_posix.py @@ -442,6 +442,14 @@ class PosixTester(unittest.TestCase): else: self.assertTrue(stat.S_ISFIFO(posix.stat(support.TESTFN).st_mode)) + # Keyword arguments are also supported + support.unlink(support.TESTFN) + try: + posix.mknod(path=support.TESTFN, mode=mode, device=0, + dir_fd=None) + except OSError as e: + self.assertIn(e.errno, (errno.EPERM, errno.EINVAL)) + @unittest.skipUnless(hasattr(posix, 'stat'), 'test needs posix.stat()') @unittest.skipUnless(hasattr(posix, 'makedev'), 'test needs posix.makedev()') def test_makedev(self): diff --git a/Lib/test/test_posixpath.py b/Lib/test/test_posixpath.py index ece3555..9d20471 100644 --- a/Lib/test/test_posixpath.py +++ b/Lib/test/test_posixpath.py @@ -316,7 +316,6 @@ class PosixPathTest(unittest.TestCase): # Bug #930024, return the path unchanged if we get into an infinite # symlink loop. try: - old_path = abspath('.') os.symlink(ABSTFN, ABSTFN) self.assertEqual(realpath(ABSTFN), ABSTFN) @@ -342,10 +341,9 @@ class PosixPathTest(unittest.TestCase): self.assertEqual(realpath(ABSTFN+"c"), ABSTFN+"c") # Test using relative path as well. - os.chdir(dirname(ABSTFN)) - self.assertEqual(realpath(basename(ABSTFN)), ABSTFN) + with support.change_cwd(dirname(ABSTFN)): + self.assertEqual(realpath(basename(ABSTFN)), ABSTFN) finally: - os.chdir(old_path) support.unlink(ABSTFN) support.unlink(ABSTFN+"1") support.unlink(ABSTFN+"2") @@ -373,7 +371,6 @@ class PosixPathTest(unittest.TestCase): @skip_if_ABSTFN_contains_backslash def test_realpath_deep_recursion(self): depth = 10 - old_path = abspath('.') try: os.mkdir(ABSTFN) for i in range(depth): @@ -382,10 +379,9 @@ class PosixPathTest(unittest.TestCase): self.assertEqual(realpath(ABSTFN + '/%d' % depth), ABSTFN) # Test using relative path as well. - os.chdir(ABSTFN) - self.assertEqual(realpath('%d' % depth), ABSTFN) + with support.change_cwd(ABSTFN): + self.assertEqual(realpath('%d' % depth), ABSTFN) finally: - os.chdir(old_path) for i in range(depth + 1): support.unlink(ABSTFN + '/%d' % i) safe_rmdir(ABSTFN) @@ -399,15 +395,13 @@ class PosixPathTest(unittest.TestCase): # /usr/doc with 'doc' being a symlink to /usr/share/doc. We call # realpath("a"). This should return /usr/share/doc/a/. try: - old_path = abspath('.') os.mkdir(ABSTFN) os.mkdir(ABSTFN + "/y") os.symlink(ABSTFN + "/y", ABSTFN + "/k") - os.chdir(ABSTFN + "/k") - self.assertEqual(realpath("a"), ABSTFN + "/y/a") + with support.change_cwd(ABSTFN + "/k"): + self.assertEqual(realpath("a"), ABSTFN + "/y/a") finally: - os.chdir(old_path) support.unlink(ABSTFN + "/k") safe_rmdir(ABSTFN + "/y") safe_rmdir(ABSTFN) @@ -424,7 +418,6 @@ class PosixPathTest(unittest.TestCase): # and a symbolic link 'link-y' pointing to 'y' in directory 'a', # then realpath("link-y/..") should return 'k', not 'a'. try: - old_path = abspath('.') os.mkdir(ABSTFN) os.mkdir(ABSTFN + "/k") os.mkdir(ABSTFN + "/k/y") @@ -433,11 +426,10 @@ class PosixPathTest(unittest.TestCase): # Absolute path. self.assertEqual(realpath(ABSTFN + "/link-y/.."), ABSTFN + "/k") # Relative path. - os.chdir(dirname(ABSTFN)) - self.assertEqual(realpath(basename(ABSTFN) + "/link-y/.."), - ABSTFN + "/k") + with support.change_cwd(dirname(ABSTFN)): + self.assertEqual(realpath(basename(ABSTFN) + "/link-y/.."), + ABSTFN + "/k") finally: - os.chdir(old_path) support.unlink(ABSTFN + "/link-y") safe_rmdir(ABSTFN + "/k/y") safe_rmdir(ABSTFN + "/k") @@ -451,17 +443,14 @@ class PosixPathTest(unittest.TestCase): # must be resolved too. try: - old_path = abspath('.') os.mkdir(ABSTFN) os.mkdir(ABSTFN + "/k") os.symlink(ABSTFN, ABSTFN + "link") - os.chdir(dirname(ABSTFN)) - - base = basename(ABSTFN) - self.assertEqual(realpath(base + "link"), ABSTFN) - self.assertEqual(realpath(base + "link/k"), ABSTFN + "/k") + with support.change_cwd(dirname(ABSTFN)): + base = basename(ABSTFN) + self.assertEqual(realpath(base + "link"), ABSTFN) + self.assertEqual(realpath(base + "link/k"), ABSTFN + "/k") finally: - os.chdir(old_path) support.unlink(ABSTFN + "link") safe_rmdir(ABSTFN + "/k") safe_rmdir(ABSTFN) diff --git a/Lib/test/test_py_compile.py b/Lib/test/test_py_compile.py index 03cca5d..4a6caa5 100644 --- a/Lib/test/test_py_compile.py +++ b/Lib/test/test_py_compile.py @@ -63,11 +63,9 @@ class PyCompileTests(unittest.TestCase): self.assertTrue(os.path.exists(self.cache_path)) def test_cwd(self): - cwd = os.getcwd() - os.chdir(self.directory) - py_compile.compile(os.path.basename(self.source_path), - os.path.basename(self.pyc_path)) - os.chdir(cwd) + with support.change_cwd(self.directory): + py_compile.compile(os.path.basename(self.source_path), + os.path.basename(self.pyc_path)) self.assertTrue(os.path.exists(self.pyc_path)) self.assertFalse(os.path.exists(self.cache_path)) diff --git a/Lib/test/test_pyexpat.py b/Lib/test/test_pyexpat.py index 08e95c6..550aebf 100644 --- a/Lib/test/test_pyexpat.py +++ b/Lib/test/test_pyexpat.py @@ -3,6 +3,7 @@ from io import BytesIO import os +import sys import sysconfig import unittest import traceback @@ -16,22 +17,47 @@ from test.support import sortdict class SetAttributeTest(unittest.TestCase): def setUp(self): self.parser = expat.ParserCreate(namespace_separator='!') - self.set_get_pairs = [ - [0, 0], - [1, 1], - [2, 1], - [0, 0], - ] + + def test_buffer_text(self): + self.assertIs(self.parser.buffer_text, False) + for x in 0, 1, 2, 0: + self.parser.buffer_text = x + self.assertIs(self.parser.buffer_text, bool(x)) + + def test_namespace_prefixes(self): + self.assertIs(self.parser.namespace_prefixes, False) + for x in 0, 1, 2, 0: + self.parser.namespace_prefixes = x + self.assertIs(self.parser.namespace_prefixes, bool(x)) def test_ordered_attributes(self): - for x, y in self.set_get_pairs: + self.assertIs(self.parser.ordered_attributes, False) + for x in 0, 1, 2, 0: self.parser.ordered_attributes = x - self.assertEqual(self.parser.ordered_attributes, y) + self.assertIs(self.parser.ordered_attributes, bool(x)) + + def test_specified_attributes(self): + self.assertIs(self.parser.specified_attributes, False) + for x in 0, 1, 2, 0: + self.parser.specified_attributes = x + self.assertIs(self.parser.specified_attributes, bool(x)) def test_specified_attributes(self): - for x, y in self.set_get_pairs: + self.assertIs(self.parser.specified_attributes, False) + for x in 0, 1, 2, 0: self.parser.specified_attributes = x - self.assertEqual(self.parser.specified_attributes, y) + self.assertIs(self.parser.specified_attributes, bool(x)) + + def test_invalid_attributes(self): + with self.assertRaises(AttributeError): + self.parser.returns_unicode = 1 + with self.assertRaises(AttributeError): + self.parser.returns_unicode + + # Issue #25019 + self.assertRaises(TypeError, setattr, self.parser, range(0xF), 0) + self.assertRaises(TypeError, self.parser.__setattr__, range(0xF), 0) + self.assertRaises(TypeError, getattr, self.parser, range(0xF)) data = b'''\ @@ -514,11 +540,14 @@ class ChardataBufferTest(unittest.TestCase): def test_wrong_size(self): parser = expat.ParserCreate() parser.buffer_text = 1 - def f(size): - parser.buffer_size = size - - self.assertRaises(ValueError, f, -1) - self.assertRaises(ValueError, f, 0) + with self.assertRaises(ValueError): + parser.buffer_size = -1 + with self.assertRaises(ValueError): + parser.buffer_size = 0 + with self.assertRaises((ValueError, OverflowError)): + parser.buffer_size = sys.maxsize + 1 + with self.assertRaises(TypeError): + parser.buffer_size = 512.0 def test_unchanged_size(self): xml1 = b"<?xml version='1.0' encoding='iso8859'?><s>" + b'a' * 512 diff --git a/Lib/test/test_shutil.py b/Lib/test/test_shutil.py index 5183c3c..522959a 100644 --- a/Lib/test/test_shutil.py +++ b/Lib/test/test_shutil.py @@ -12,11 +12,9 @@ import errno import functools import subprocess from contextlib import ExitStack -from test import support -from test.support import TESTFN from os.path import splitdrive from distutils.spawn import find_executable, spawn -from shutil import (_make_tarball, _make_zipfile, make_archive, +from shutil import (make_archive, register_archive_format, unregister_archive_format, get_archive_formats, Error, unpack_archive, register_unpack_format, RegistryError, @@ -94,6 +92,18 @@ def read_file(path, binary=False): with open(path, 'rb' if binary else 'r') as fp: return fp.read() +def rlistdir(path): + res = [] + for name in sorted(os.listdir(path)): + p = os.path.join(path, name) + if os.path.isdir(p) and not os.path.islink(p): + res.append(name + '/') + for n in rlistdir(p): + res.append(name + '/' + n) + else: + res.append(name) + return res + class TestShutil(unittest.TestCase): @@ -959,139 +969,143 @@ class TestShutil(unittest.TestCase): @requires_zlib def test_make_tarball(self): # creating something to tar - tmpdir = self.mkdtemp() - write_file((tmpdir, 'file1'), 'xxx') - write_file((tmpdir, 'file2'), 'xxx') - os.mkdir(os.path.join(tmpdir, 'sub')) - write_file((tmpdir, 'sub', 'file3'), 'xxx') + root_dir, base_dir = self._create_files('') tmpdir2 = self.mkdtemp() # force shutil to create the directory os.rmdir(tmpdir2) - unittest.skipUnless(splitdrive(tmpdir)[0] == splitdrive(tmpdir2)[0], - "source and target should be on same drive") + # working with relative paths + work_dir = os.path.dirname(tmpdir2) + rel_base_name = os.path.join(os.path.basename(tmpdir2), 'archive') - base_name = os.path.join(tmpdir2, 'archive') - - # working with relative paths to avoid tar warnings - old_dir = os.getcwd() - os.chdir(tmpdir) - try: - _make_tarball(splitdrive(base_name)[1], '.') - finally: - os.chdir(old_dir) + with support.change_cwd(work_dir): + base_name = os.path.abspath(rel_base_name) + tarball = make_archive(rel_base_name, 'gztar', root_dir, '.') # check if the compressed tarball was created - tarball = base_name + '.tar.gz' - self.assertTrue(os.path.exists(tarball)) + self.assertEqual(tarball, base_name + '.tar.gz') + self.assertTrue(os.path.isfile(tarball)) + self.assertTrue(tarfile.is_tarfile(tarball)) + with tarfile.open(tarball, 'r:gz') as tf: + self.assertCountEqual(tf.getnames(), + ['.', './sub', './sub2', + './file1', './file2', './sub/file3']) # trying an uncompressed one - base_name = os.path.join(tmpdir2, 'archive') - old_dir = os.getcwd() - os.chdir(tmpdir) - try: - _make_tarball(splitdrive(base_name)[1], '.', compress=None) - finally: - os.chdir(old_dir) - tarball = base_name + '.tar' - self.assertTrue(os.path.exists(tarball)) + with support.change_cwd(work_dir): + tarball = make_archive(rel_base_name, 'tar', root_dir, '.') + self.assertEqual(tarball, base_name + '.tar') + self.assertTrue(os.path.isfile(tarball)) + self.assertTrue(tarfile.is_tarfile(tarball)) + with tarfile.open(tarball, 'r') as tf: + self.assertCountEqual(tf.getnames(), + ['.', './sub', './sub2', + './file1', './file2', './sub/file3']) def _tarinfo(self, path): - tar = tarfile.open(path) - try: + with tarfile.open(path) as tar: names = tar.getnames() names.sort() return tuple(names) - finally: - tar.close() - def _create_files(self): + def _create_files(self, base_dir='dist'): # creating something to tar - tmpdir = self.mkdtemp() - dist = os.path.join(tmpdir, 'dist') - os.mkdir(dist) + root_dir = self.mkdtemp() + dist = os.path.join(root_dir, base_dir) + os.makedirs(dist, exist_ok=True) write_file((dist, 'file1'), 'xxx') write_file((dist, 'file2'), 'xxx') os.mkdir(os.path.join(dist, 'sub')) write_file((dist, 'sub', 'file3'), 'xxx') os.mkdir(os.path.join(dist, 'sub2')) - tmpdir2 = self.mkdtemp() - base_name = os.path.join(tmpdir2, 'archive') - return tmpdir, tmpdir2, base_name + if base_dir: + write_file((root_dir, 'outer'), 'xxx') + return root_dir, base_dir @requires_zlib - @unittest.skipUnless(find_executable('tar') and find_executable('gzip'), + @unittest.skipUnless(find_executable('tar'), 'Need the tar command to run') def test_tarfile_vs_tar(self): - tmpdir, tmpdir2, base_name = self._create_files() - old_dir = os.getcwd() - os.chdir(tmpdir) - try: - _make_tarball(base_name, 'dist') - finally: - os.chdir(old_dir) + root_dir, base_dir = self._create_files() + base_name = os.path.join(self.mkdtemp(), 'archive') + tarball = make_archive(base_name, 'gztar', root_dir, base_dir) # check if the compressed tarball was created - tarball = base_name + '.tar.gz' - self.assertTrue(os.path.exists(tarball)) + self.assertEqual(tarball, base_name + '.tar.gz') + self.assertTrue(os.path.isfile(tarball)) # now create another tarball using `tar` - tarball2 = os.path.join(tmpdir, 'archive2.tar.gz') - tar_cmd = ['tar', '-cf', 'archive2.tar', 'dist'] - gzip_cmd = ['gzip', '-f9', 'archive2.tar'] - old_dir = os.getcwd() - os.chdir(tmpdir) - try: - with captured_stdout() as s: - spawn(tar_cmd) - spawn(gzip_cmd) - finally: - os.chdir(old_dir) + tarball2 = os.path.join(root_dir, 'archive2.tar') + tar_cmd = ['tar', '-cf', 'archive2.tar', base_dir] + with support.change_cwd(root_dir), captured_stdout(): + spawn(tar_cmd) - self.assertTrue(os.path.exists(tarball2)) + self.assertTrue(os.path.isfile(tarball2)) # let's compare both tarballs self.assertEqual(self._tarinfo(tarball), self._tarinfo(tarball2)) # trying an uncompressed one - base_name = os.path.join(tmpdir2, 'archive') - old_dir = os.getcwd() - os.chdir(tmpdir) - try: - _make_tarball(base_name, 'dist', compress=None) - finally: - os.chdir(old_dir) - tarball = base_name + '.tar' - self.assertTrue(os.path.exists(tarball)) + tarball = make_archive(base_name, 'tar', root_dir, base_dir) + self.assertEqual(tarball, base_name + '.tar') + self.assertTrue(os.path.isfile(tarball)) # now for a dry_run - base_name = os.path.join(tmpdir2, 'archive') - old_dir = os.getcwd() - os.chdir(tmpdir) - try: - _make_tarball(base_name, 'dist', compress=None, dry_run=True) - finally: - os.chdir(old_dir) - tarball = base_name + '.tar' - self.assertTrue(os.path.exists(tarball)) + tarball = make_archive(base_name, 'tar', root_dir, base_dir, + dry_run=True) + self.assertEqual(tarball, base_name + '.tar') + self.assertTrue(os.path.isfile(tarball)) @requires_zlib @unittest.skipUnless(ZIP_SUPPORT, 'Need zip support to run') def test_make_zipfile(self): - # creating something to tar - tmpdir = self.mkdtemp() - write_file((tmpdir, 'file1'), 'xxx') - write_file((tmpdir, 'file2'), 'xxx') + # creating something to zip + root_dir, base_dir = self._create_files() tmpdir2 = self.mkdtemp() # force shutil to create the directory os.rmdir(tmpdir2) - base_name = os.path.join(tmpdir2, 'archive') - _make_zipfile(base_name, tmpdir) + # working with relative paths + work_dir = os.path.dirname(tmpdir2) + rel_base_name = os.path.join(os.path.basename(tmpdir2), 'archive') + + with support.change_cwd(work_dir): + base_name = os.path.abspath(rel_base_name) + res = make_archive(rel_base_name, 'zip', root_dir, base_dir) + + self.assertEqual(res, base_name + '.zip') + self.assertTrue(os.path.isfile(res)) + self.assertTrue(zipfile.is_zipfile(res)) + with zipfile.ZipFile(res) as zf: + self.assertCountEqual(zf.namelist(), + ['dist/', 'dist/sub/', 'dist/sub2/', + 'dist/file1', 'dist/file2', 'dist/sub/file3']) - # check if the compressed tarball was created - tarball = base_name + '.zip' - self.assertTrue(os.path.exists(tarball)) + @requires_zlib + @unittest.skipUnless(ZIP_SUPPORT, 'Need zip support to run') + @unittest.skipUnless(find_executable('zip'), + 'Need the zip command to run') + def test_zipfile_vs_zip(self): + root_dir, base_dir = self._create_files() + base_name = os.path.join(self.mkdtemp(), 'archive') + archive = make_archive(base_name, 'zip', root_dir, base_dir) + + # check if ZIP file was created + self.assertEqual(archive, base_name + '.zip') + self.assertTrue(os.path.isfile(archive)) + + # now create another ZIP file using `zip` + archive2 = os.path.join(root_dir, 'archive2.zip') + zip_cmd = ['zip', '-q', '-r', 'archive2.zip', base_dir] + with support.change_cwd(root_dir): + spawn(zip_cmd) + self.assertTrue(os.path.isfile(archive2)) + # let's compare both ZIP files + with zipfile.ZipFile(archive) as zf: + names = zf.namelist() + with zipfile.ZipFile(archive2) as zf: + names2 = zf.namelist() + self.assertEqual(sorted(names), sorted(names2)) def test_make_archive(self): tmpdir = self.mkdtemp() @@ -1108,40 +1122,37 @@ class TestShutil(unittest.TestCase): else: group = owner = 'root' - base_dir, root_dir, base_name = self._create_files() - base_name = os.path.join(self.mkdtemp() , 'archive') + root_dir, base_dir = self._create_files() + base_name = os.path.join(self.mkdtemp(), 'archive') res = make_archive(base_name, 'zip', root_dir, base_dir, owner=owner, group=group) - self.assertTrue(os.path.exists(res)) + self.assertTrue(os.path.isfile(res)) res = make_archive(base_name, 'zip', root_dir, base_dir) - self.assertTrue(os.path.exists(res)) + self.assertTrue(os.path.isfile(res)) res = make_archive(base_name, 'tar', root_dir, base_dir, owner=owner, group=group) - self.assertTrue(os.path.exists(res)) + self.assertTrue(os.path.isfile(res)) res = make_archive(base_name, 'tar', root_dir, base_dir, owner='kjhkjhkjg', group='oihohoh') - self.assertTrue(os.path.exists(res)) + self.assertTrue(os.path.isfile(res)) @requires_zlib @unittest.skipUnless(UID_GID_SUPPORT, "Requires grp and pwd support") def test_tarfile_root_owner(self): - tmpdir, tmpdir2, base_name = self._create_files() - old_dir = os.getcwd() - os.chdir(tmpdir) + root_dir, base_dir = self._create_files() + base_name = os.path.join(self.mkdtemp(), 'archive') group = grp.getgrgid(0)[0] owner = pwd.getpwuid(0)[0] - try: - archive_name = _make_tarball(base_name, 'dist', compress=None, - owner=owner, group=group) - finally: - os.chdir(old_dir) + with support.change_cwd(root_dir): + archive_name = make_archive(base_name, 'gztar', root_dir, 'dist', + owner=owner, group=group) # check if the compressed tarball was created - self.assertTrue(os.path.exists(archive_name)) + self.assertTrue(os.path.isfile(archive_name)) # now checks the rights archive = tarfile.open(archive_name) @@ -1198,18 +1209,6 @@ class TestShutil(unittest.TestCase): formats = [name for name, params in get_archive_formats()] self.assertNotIn('xxx', formats) - def _compare_dirs(self, dir1, dir2): - # check that dir1 and dir2 are equivalent, - # return the diff - diff = [] - for root, dirs, files in os.walk(dir1): - for file_ in files: - path = os.path.join(root, file_) - target_path = os.path.join(dir2, os.path.split(path)[-1]) - if not os.path.exists(target_path): - diff.append(file_) - return diff - @requires_zlib def test_unpack_archive(self): formats = ['tar', 'gztar', 'zip'] @@ -1218,22 +1217,22 @@ class TestShutil(unittest.TestCase): if LZMA_SUPPORTED: formats.append('xztar') + root_dir, base_dir = self._create_files() + expected = rlistdir(root_dir) + expected.remove('outer') for format in formats: - tmpdir = self.mkdtemp() - base_dir, root_dir, base_name = self._create_files() - tmpdir2 = self.mkdtemp() + base_name = os.path.join(self.mkdtemp(), 'archive') filename = make_archive(base_name, format, root_dir, base_dir) # let's try to unpack it now + tmpdir2 = self.mkdtemp() unpack_archive(filename, tmpdir2) - diff = self._compare_dirs(tmpdir, tmpdir2) - self.assertEqual(diff, []) + self.assertEqual(rlistdir(tmpdir2), expected) # and again, this time with the format specified tmpdir3 = self.mkdtemp() unpack_archive(filename, tmpdir3, format=format) - diff = self._compare_dirs(tmpdir, tmpdir3) - self.assertEqual(diff, []) + self.assertEqual(rlistdir(tmpdir3), expected) self.assertRaises(shutil.ReadError, unpack_archive, TESTFN) self.assertRaises(ValueError, unpack_archive, TESTFN, format='xxx') diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py index 9c0229a..ff2010d 100644 --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -317,11 +317,8 @@ class ProcessTestCase(BaseTestCase): # Normalize an expected cwd (for Tru64 support). # We can't use os.path.realpath since it doesn't expand Tru64 {memb} # strings. See bug #1063571. - original_cwd = os.getcwd() - os.chdir(cwd) - cwd = os.getcwd() - os.chdir(original_cwd) - return cwd + with support.change_cwd(cwd): + return os.getcwd() # For use in the test_cwd* tests below. def _split_python_path(self): diff --git a/Lib/test/test_sysconfig.py b/Lib/test/test_sysconfig.py index c0f27a6..0917c3e 100644 --- a/Lib/test/test_sysconfig.py +++ b/Lib/test/test_sysconfig.py @@ -6,7 +6,7 @@ import shutil from copy import copy from test.support import (run_unittest, TESTFN, unlink, check_warnings, - captured_stdout, skip_unless_symlink) + captured_stdout, skip_unless_symlink, change_cwd) import sysconfig from sysconfig import (get_paths, get_platform, get_config_vars, @@ -361,12 +361,8 @@ class TestSysConfig(unittest.TestCase): # srcdir should be independent of the current working directory # See Issues #15322, #15364. srcdir = sysconfig.get_config_var('srcdir') - cwd = os.getcwd() - try: - os.chdir('..') + with change_cwd(os.pardir): srcdir2 = sysconfig.get_config_var('srcdir') - finally: - os.chdir(cwd) self.assertEqual(srcdir, srcdir2) @unittest.skipIf(sysconfig.get_config_var('EXT_SUFFIX') is None, diff --git a/Lib/test/test_tarfile.py b/Lib/test/test_tarfile.py index 57cddb2..1412cae 100644 --- a/Lib/test/test_tarfile.py +++ b/Lib/test/test_tarfile.py @@ -1132,10 +1132,8 @@ class WriteTest(WriteTestBase, unittest.TestCase): self.assertEqual(tar.getnames(), [], "added the archive to itself") - cwd = os.getcwd() - os.chdir(TEMPDIR) - tar.add(dstname) - os.chdir(cwd) + with support.change_cwd(TEMPDIR): + tar.add(dstname) self.assertEqual(tar.getnames(), [], "added the archive to itself") finally: @@ -1292,9 +1290,7 @@ class WriteTest(WriteTestBase, unittest.TestCase): def test_cwd(self): # Test adding the current working directory. - cwd = os.getcwd() - os.chdir(TEMPDIR) - try: + with support.change_cwd(TEMPDIR): tar = tarfile.open(tmpname, self.mode) try: tar.add(".") @@ -1308,8 +1304,6 @@ class WriteTest(WriteTestBase, unittest.TestCase): self.assertTrue(t.name.startswith("./"), t.name) finally: tar.close() - finally: - os.chdir(cwd) def test_open_nonwritable_fileobj(self): for exctype in OSError, EOFError, RuntimeError: diff --git a/Lib/test/test_unicode_file.py b/Lib/test/test_unicode_file.py index faa8da3..e4709a1 100644 --- a/Lib/test/test_unicode_file.py +++ b/Lib/test/test_unicode_file.py @@ -5,7 +5,7 @@ import os, glob, time, shutil import unicodedata import unittest -from test.support import (run_unittest, rmtree, +from test.support import (run_unittest, rmtree, change_cwd, TESTFN_ENCODING, TESTFN_UNICODE, TESTFN_UNENCODABLE, create_empty_file) if not os.path.supports_unicode_filenames: @@ -82,13 +82,11 @@ class TestUnicodeFiles(unittest.TestCase): self.assertFalse(os.path.exists(filename2 + '.new')) def _do_directory(self, make_name, chdir_name): - cwd = os.getcwd() if os.path.isdir(make_name): rmtree(make_name) os.mkdir(make_name) try: - os.chdir(chdir_name) - try: + with change_cwd(chdir_name): cwd_result = os.getcwd() name_result = make_name @@ -96,8 +94,6 @@ class TestUnicodeFiles(unittest.TestCase): name_result = unicodedata.normalize("NFD", name_result) self.assertEqual(os.path.basename(cwd_result),name_result) - finally: - os.chdir(cwd) finally: os.rmdir(make_name) diff --git a/Lib/test/test_warnings/__init__.py b/Lib/test/test_warnings/__init__.py index 991a249..cea9c57 100644 --- a/Lib/test/test_warnings/__init__.py +++ b/Lib/test/test_warnings/__init__.py @@ -44,6 +44,7 @@ class BaseTest: """Basic bookkeeping required for testing.""" def setUp(self): + self.old_unittest_module = unittest.case.warnings # The __warningregistry__ needs to be in a pristine state for tests # to work properly. if '__warningregistry__' in globals(): @@ -55,10 +56,15 @@ class BaseTest: # The 'warnings' module must be explicitly set so that the proper # interaction between _warnings and 'warnings' can be controlled. sys.modules['warnings'] = self.module + # Ensure that unittest.TestCase.assertWarns() uses the same warnings + # module than warnings.catch_warnings(). Otherwise, + # warnings.catch_warnings() will be unable to remove the added filter. + unittest.case.warnings = self.module super(BaseTest, self).setUp() def tearDown(self): sys.modules['warnings'] = original_warnings + unittest.case.warnings = self.old_unittest_module super(BaseTest, self).tearDown() class PublicAPITests(BaseTest): diff --git a/Lib/test/test_wsgiref.py b/Lib/test/test_wsgiref.py index 112a1b9..8cca595 100644 --- a/Lib/test/test_wsgiref.py +++ b/Lib/test/test_wsgiref.py @@ -1,11 +1,10 @@ -from __future__ import nested_scopes # Backward compat for 2.1 from unittest import TestCase from wsgiref.util import setup_testing_defaults from wsgiref.headers import Headers from wsgiref.handlers import BaseHandler, BaseCGIHandler from wsgiref import util from wsgiref.validate import validator -from wsgiref.simple_server import WSGIServer, WSGIRequestHandler, demo_app +from wsgiref.simple_server import WSGIServer, WSGIRequestHandler from wsgiref.simple_server import make_server from io import StringIO, BytesIO, BufferedReader from socketserver import BaseServer @@ -14,8 +13,8 @@ from platform import python_implementation import os import re import sys +import unittest -from test import support class MockServer(WSGIServer): """Non-socket HTTP server""" diff --git a/Lib/test/test_zlib.py b/Lib/test/test_zlib.py index bd935ed..7cd1d7c 100644 --- a/Lib/test/test_zlib.py +++ b/Lib/test/test_zlib.py @@ -222,9 +222,9 @@ class CompressObjectTestCase(BaseCompressTestCase, unittest.TestCase): level = 2 method = zlib.DEFLATED wbits = -12 - memlevel = 9 + memLevel = 9 strategy = zlib.Z_FILTERED - co = zlib.compressobj(level, method, wbits, memlevel, strategy) + co = zlib.compressobj(level, method, wbits, memLevel, strategy) x1 = co.compress(HAMLET_SCENE) x2 = co.flush() dco = zlib.decompressobj(wbits) @@ -232,6 +232,10 @@ class CompressObjectTestCase(BaseCompressTestCase, unittest.TestCase): y2 = dco.flush() self.assertEqual(HAMLET_SCENE, y1 + y2) + # keyword arguments should also be supported + zlib.compressobj(level=level, method=method, wbits=wbits, + memLevel=memLevel, strategy=strategy, zdict=b"") + def test_compressincremental(self): # compress object in steps, decompress object as one-shot data = HAMLET_SCENE * 128 diff --git a/Lib/unittest/case.py b/Lib/unittest/case.py index 7701ad3..ac8d67d 100644 --- a/Lib/unittest/case.py +++ b/Lib/unittest/case.py @@ -583,8 +583,11 @@ class TestCase(object): finally: result.stopTest(self) return - expecting_failure = getattr(testMethod, - "__unittest_expecting_failure__", False) + expecting_failure_method = getattr(testMethod, + "__unittest_expecting_failure__", False) + expecting_failure_class = getattr(self, + "__unittest_expecting_failure__", False) + expecting_failure = expecting_failure_class or expecting_failure_method outcome = _Outcome(result) try: self._outcome = outcome @@ -1279,8 +1282,10 @@ class TestCase(object): assert expected_regex, "expected_regex must not be empty." expected_regex = re.compile(expected_regex) if not expected_regex.search(text): - msg = msg or "Regex didn't match" - msg = '%s: %r not found in %r' % (msg, expected_regex.pattern, text) + standardMsg = "Regex didn't match: %r not found in %r" % ( + expected_regex.pattern, text) + # _formatMessage ensures the longMessage option is respected + msg = self._formatMessage(msg, standardMsg) raise self.failureException(msg) def assertNotRegex(self, text, unexpected_regex, msg=None): @@ -1289,11 +1294,12 @@ class TestCase(object): unexpected_regex = re.compile(unexpected_regex) match = unexpected_regex.search(text) if match: - msg = msg or "Regex matched" - msg = '%s: %r matches %r in %r' % (msg, - text[match.start():match.end()], - unexpected_regex.pattern, - text) + standardMsg = 'Regex matched: %r matches %r in %r' % ( + text[match.start() : match.end()], + unexpected_regex.pattern, + text) + # _formatMessage ensures the longMessage option is respected + msg = self._formatMessage(msg, standardMsg) raise self.failureException(msg) @@ -1315,6 +1321,7 @@ class TestCase(object): failIf = _deprecate(assertFalse) assertRaisesRegexp = _deprecate(assertRaisesRegex) assertRegexpMatches = _deprecate(assertRegex) + assertNotRegexpMatches = _deprecate(assertNotRegex) diff --git a/Lib/unittest/mock.py b/Lib/unittest/mock.py index 4c58639..99ce1e2 100644 --- a/Lib/unittest/mock.py +++ b/Lib/unittest/mock.py @@ -2005,8 +2005,7 @@ class _Call(tuple): else: other_args = () other_kwargs = value - else: - # len 2 + elif len_other == 2: # could be (name, args) or (name, kwargs) or (args, kwargs) first, second = other if isinstance(first, str): @@ -2017,6 +2016,8 @@ class _Call(tuple): other_args, other_kwargs = (), second else: other_args, other_kwargs = first, second + else: + return False if self_name and other_name != self_name: return False diff --git a/Lib/unittest/test/test_assertions.py b/Lib/unittest/test/test_assertions.py index c349a95..e6e2bc2 100644 --- a/Lib/unittest/test/test_assertions.py +++ b/Lib/unittest/test/test_assertions.py @@ -133,7 +133,6 @@ class Test_Assertions(unittest.TestCase): try: self.assertNotRegex('Ala ma kota', r'k.t', 'Message') except self.failureException as e: - self.assertIn("'kot'", e.args[0]) self.assertIn('Message', e.args[0]) else: self.fail('assertNotRegex should have failed.') @@ -329,6 +328,20 @@ class TestLongMessage(unittest.TestCase): "^unexpectedly identical: None$", "^unexpectedly identical: None : oops$"]) + def testAssertRegex(self): + self.assertMessages('assertRegex', ('foo', 'bar'), + ["^Regex didn't match:", + "^oops$", + "^Regex didn't match:", + "^Regex didn't match: (.*) : oops$"]) + + def testAssertNotRegex(self): + self.assertMessages('assertNotRegex', ('foo', 'foo'), + ["^Regex matched:", + "^oops$", + "^Regex matched:", + "^Regex matched: (.*) : oops$"]) + def assertMessagesCM(self, methodName, args, func, errors): """ diff --git a/Lib/unittest/test/test_skipping.py b/Lib/unittest/test/test_skipping.py index 807510f..71f7b70 100644 --- a/Lib/unittest/test/test_skipping.py +++ b/Lib/unittest/test/test_skipping.py @@ -120,6 +120,39 @@ class Test_TestSkipping(unittest.TestCase): self.assertEqual(result.expectedFailures[0][0], test) self.assertTrue(result.wasSuccessful()) + def test_expected_failure_with_wrapped_class(self): + @unittest.expectedFailure + class Foo(unittest.TestCase): + def test_1(self): + self.assertTrue(False) + + events = [] + result = LoggingResult(events) + test = Foo("test_1") + test.run(result) + self.assertEqual(events, + ['startTest', 'addExpectedFailure', 'stopTest']) + self.assertEqual(result.expectedFailures[0][0], test) + self.assertTrue(result.wasSuccessful()) + + def test_expected_failure_with_wrapped_subclass(self): + class Foo(unittest.TestCase): + def test_1(self): + self.assertTrue(False) + + @unittest.expectedFailure + class Bar(Foo): + pass + + events = [] + result = LoggingResult(events) + test = Bar("test_1") + test.run(result) + self.assertEqual(events, + ['startTest', 'addExpectedFailure', 'stopTest']) + self.assertEqual(result.expectedFailures[0][0], test) + self.assertTrue(result.wasSuccessful()) + def test_expected_failure_subtests(self): # A failure in any subtest counts as the expected failure of the # whole test. diff --git a/Lib/unittest/test/testmock/testmock.py b/Lib/unittest/test/testmock/testmock.py index 97d844f..2a6069f 100644 --- a/Lib/unittest/test/testmock/testmock.py +++ b/Lib/unittest/test/testmock/testmock.py @@ -300,6 +300,9 @@ class MockTest(unittest.TestCase): self.assertEqual(mock.call_args, ((sentinel.Arg,), {"kw": sentinel.Kwarg})) + # Comparing call_args to a long sequence should not raise + # an exception. See issue 24857. + self.assertFalse(mock.call_args == "a long sequence") def test_assert_called_with(self): mock = Mock() @@ -103,6 +103,7 @@ Mike Bayer Samuel L. Bayer Donald Beaudry David Beazley +John Beck Ingolf Becker Neal Becker Robin Becker @@ -622,6 +623,7 @@ Ken Howard Brad Howes Mike Hoy Ben Hoyt +Chiu-Hsiang Hsu Chih-Hao Huang Christian Hudon Lawrence Hudson @@ -785,6 +787,7 @@ Andrew Kuchling Dave Kuhlman Jon Kuhn Toshio Kuratomi +Ilia Kurenkov Vladimir Kushnir Erno Kuusela Ross Lagerwall @@ -794,6 +797,7 @@ Thomas Lamb Valerie Lambert Jean-Baptiste "Jiba" Lamy Ronan Lamy +Peter Landry Torsten Landschoff Łukasz Langa Tino Lange @@ -827,6 +831,7 @@ Joerg Lehmann Robert Lehmann Petri Lehtinen Luke Kenneth Casson Leighton +John Leitch Tshepang Lekhonkhobe Marc-André Lemburg Mateusz Lenik @@ -911,12 +916,14 @@ Nick Mathewson Simon Mathieu Laura Matson Graham Matthews +mattip Martin Matusiak Dieter Maurer Daniel May Madison May Lucas Maystre Arnaud Mazin +Pam McA'Nulty Matt McClure Jack McCracken Rebecca McCreary @@ -1061,6 +1068,7 @@ Jan Palus Yongzhi Pan Martin Panter Mathias Panzenböck +Marco Paolini M. Papillon Peter Parente Alexandre Parenteau @@ -1121,6 +1129,7 @@ Martin Pool Iustin Pop Claudiu Popa John Popplewell +Matheus Vieira Portela Davin Potts Guillaume Pratte Florian Preinstorfer @@ -1182,6 +1191,7 @@ Vlad Riscutia Wes Rishel Daniel Riti Juan M. Bello Rivas +Mohd Sanad Zaki Rizvi Davide Rizzo Anthony Roach Carl Robben @@ -1302,6 +1312,7 @@ Alexander Shigin Pete Shinners Michael Shiplett John W. Shipman +Shiyao Ma Alex Shkop Joel Shprentz Yue Shuaijie @@ -1414,6 +1425,7 @@ Eric Tiedemann July Tikhonov Tracy Tims Oren Tirosh +Tim Tisdall Jason Tishler Christian Tismer Jim Tittsler @@ -1514,6 +1526,7 @@ Bob Weiner Edward Welbourne Cliff Wells Rickard Westman +Joseph Weston Jeff Wheeler Christopher White David White @@ -2,10 +2,107 @@ Python News +++++++++++ -What's New in Python 3.5.0 final? -================================= -Release date: 2015-09-13 +What's New in Python 3.5.1 +========================== + +Release date: TBA + +Core and Builtins +----------------- + +Library +------- + +- Issue #24684: socket.socket.getaddrinfo() now calls + PyUnicode_AsEncodedString() instead of calling the encode() method of the + host, to handle correctly custom string with an encode() method which doesn't + return a byte string. The encoder of the IDNA codec is now called directly + instead of calling the encode() method of the string. + +- Issue #25060: Correctly compute stack usage of the BUILD_MAP opcode. + +- Issue #24857: Comparing call_args to a long sequence now correctly returns a + boolean result instead of raising an exception. Patch by A Kaptur. + +- Issue #23144: Make sure that HTMLParser.feed() returns all the data, even + when convert_charrefs is True. + +- Issue #24982: shutil.make_archive() with the "zip" format now adds entries + for directories (including empty directories) in ZIP file. + +- Issue #25019: Fixed a crash caused by setting non-string key of expat parser. + Based on patch by John Leitch. + +- Issue #16180: Exit pdb if file has syntax error, instead of trapping user + in an infinite loop. Patch by Xavier de Gaye. + +- Issue #24891: Fix a race condition at Python startup if the file descriptor + of stdin (0), stdout (1) or stderr (2) is closed while Python is creating + sys.stdin, sys.stdout and sys.stderr objects. These attributes are now set + to None if the creation of the object failed, instead of raising an OSError + exception. Initial patch written by Marco Paolini. + +- Issue #24992: Fix error handling and a race condition (related to garbage + collection) in collections.OrderedDict constructor. + +- Issue #24881: Fixed setting binary mode in Python implementation of FileIO + on Windows and Cygwin. Patch from Akira Li. + +- Issue #21112: Fix regression in unittest.expectedFailure on subclasses. + Patch from Berker Peksag. + +- Issue #24764: cgi.FieldStorage.read_multi() now ignores the Content-Length + header in part headers. Patch written by Peter Landry and reviewed by Pierre + Quentel. + +- Issue #24913: Fix overrun error in deque.index(). + Found by John Leitch and Bryce Darling. + +- Issue #24774: Fix docstring in http.server.test. Patch from Chiu-Hsiang Hsu. + +- Issue #21159: Improve message in configparser.InterpolationMissingOptionError. + Patch from Łukasz Langa. + +- Issue #20362: Honour TestCase.longMessage correctly in assertRegex. + Patch from Ilia Kurenkov. + +- Issue #23572: Fixed functools.singledispatch on classes with falsy + metaclasses. Patch by Ethan Furman. + +Documentation +------------- + +- Issue #24952: Clarify the default size argument of stack_size() in + the "threading" and "_thread" modules. Patch from Mattip. + +- Issue #23725: Overhaul tempfile docs. Note deprecated status of mktemp. + Patch from Zbigniew Jędrzejewski-Szmek. + +- Issue #24808: Update the types of some PyTypeObject fields. Patch by + Joseph Weston. + +- Issue #22812: Fix unittest discovery examples. + Patch from Pam McA'Nulty. + +Tests +----- + +- PCbuild\rt.bat now accepts an unlimited number of arguments to pass along + to regrtest.py. Previously there was a limit of 9. + +Build +----- + +- Issue #24910: Windows MSIs now have unique display names. + +- Issue #24986: It is now possible to build Python on Windows without errors + when external libraries are not available. + +Windows +------- + +- Issue #25022: Removed very outdated PC/example_nt/ directory. Build ----- diff --git a/Misc/Porting b/Misc/Porting index c283cb2..c43b112 100644 --- a/Misc/Porting +++ b/Misc/Porting @@ -1,41 +1 @@ -Q. I want to port Python to a new platform. How do I begin? - -A. I guess the two things to start with is to familiarize yourself -with are the development system for your target platform and the -generic build process for Python. Make sure you can compile and run a -simple hello-world program on your target platform. Make sure you can -compile and run the Python interpreter on a platform to which it has -already been ported (preferably Unix, but Mac or Windows will do, -too). - -I also would never start something like this without at least -medium-level understanding of your target platform (i.e. how it is -generally used, how to write platform specific apps etc.) and Python -(or else you'll never know how to test the results). - -The build process for Python, in particular the Makefiles in the -source distribution, will give you a hint on which files to compile -for Python. Not all source files are relevant -- some are platform -specific, others are only used in emergencies (e.g. getopt.c). The -Makefiles tell the story. - -You'll also need a pyconfig.h file tailored for your platform. You can -start with pyconfig.h.in, read the comments and turn on definitions that -apply to your platform. - -And you'll need a config.c file, which lists the built-in modules you -support. Start with Modules/config.c.in. - -Finally, you'll run into some things that aren't supported on your -target platform. Forget about the posix module for now -- simply take -it out of the config.c file. - -Bang on it until you get a >>> prompt. (You may have to disable the -importing of "site.py" by passing the -S option.) - -Then bang on it until it executes very simple Python statements. - -Now bang on it some more. At some point you'll want to use the os -module; this is the time to start thinking about what to do with the -posix module. It's okay to simply #ifdef out those functions that -cause problems; the remaining ones will be quite useful. +This document is moved to https://docs.python.org/devguide/faq.html#how-do-i-port-python-to-a-new-platform diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c index 3856d83..f450f25 100644 --- a/Modules/_collectionsmodule.c +++ b/Modules/_collectionsmodule.c @@ -419,9 +419,11 @@ deque_extend(dequeobject *deque, PyObject *iterable) deque->rightblock->data[deque->rightindex] = item; deque_trim_left(deque); } - Py_DECREF(it); - if (PyErr_Occurred()) + if (PyErr_Occurred()) { + Py_DECREF(it); return NULL; + } + Py_DECREF(it); Py_RETURN_NONE; } @@ -480,9 +482,11 @@ deque_extendleft(dequeobject *deque, PyObject *iterable) deque->leftblock->data[deque->leftindex] = item; deque_trim_right(deque); } - Py_DECREF(it); - if (PyErr_Occurred()) + if (PyErr_Occurred()) { + Py_DECREF(it); return NULL; + } + Py_DECREF(it); Py_RETURN_NONE; } @@ -497,8 +501,8 @@ deque_inplace_concat(dequeobject *deque, PyObject *other) result = deque_extend(deque, other); if (result == NULL) return result; - Py_DECREF(result); Py_INCREF(deque); + Py_DECREF(result); return (PyObject *)deque; } @@ -1260,8 +1264,8 @@ deque_repr(PyObject *deque) aslist, ((dequeobject *)deque)->maxlen); else result = PyUnicode_FromFormat("deque(%R)", aslist); - Py_DECREF(aslist); Py_ReprLeave(deque); + Py_DECREF(aslist); return result; } diff --git a/Modules/_decimal/libmpdec/mpdecimal.c b/Modules/_decimal/libmpdec/mpdecimal.c index 21d2222..593f9f5 100644 --- a/Modules/_decimal/libmpdec/mpdecimal.c +++ b/Modules/_decimal/libmpdec/mpdecimal.c @@ -43,6 +43,7 @@ #ifdef PPRO #if defined(_MSC_VER) #include <float.h> + #pragma float_control(precise, on) #pragma fenv_access(on) #elif !defined(__OpenBSD__) && !defined(__NetBSD__) /* C99 */ diff --git a/Modules/clinic/posixmodule.c.h b/Modules/clinic/posixmodule.c.h index 98eeae4..9ef702a 100644 --- a/Modules/clinic/posixmodule.c.h +++ b/Modules/clinic/posixmodule.c.h @@ -1487,10 +1487,10 @@ PyDoc_STRVAR(os_utime__doc__, "\n" "If times is not None, it must be a tuple (atime, mtime);\n" " atime and mtime should be expressed as float seconds since the epoch.\n" -"If ns is not None, it must be a tuple (atime_ns, mtime_ns);\n" +"If ns is specified, it must be a tuple (atime_ns, mtime_ns);\n" " atime_ns and mtime_ns should be expressed as integer nanoseconds\n" " since the epoch.\n" -"If both times and ns are None, utime uses the current time.\n" +"If times is None and ns is unspecified, utime uses the current time.\n" "Specifying tuples for both times and ns is an error.\n" "\n" "If dir_fd is not None, it should be a file descriptor open to a directory,\n" @@ -5788,4 +5788,4 @@ exit: #ifndef OS_SET_HANDLE_INHERITABLE_METHODDEF #define OS_SET_HANDLE_INHERITABLE_METHODDEF #endif /* !defined(OS_SET_HANDLE_INHERITABLE_METHODDEF) */ -/*[clinic end generated code: output=f3f92b2d2e2c3fe3 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=95824c52fd034654 input=a9049054013a1b77]*/ diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c index ec8c526..c0baf6a 100644 --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -4560,9 +4560,7 @@ typedef struct { } \ -#define UTIME_HAVE_DIR_FD (defined(HAVE_FUTIMESAT) || defined(HAVE_UTIMENSAT)) - -#if UTIME_HAVE_DIR_FD +#if defined(HAVE_FUTIMESAT) || defined(HAVE_UTIMENSAT) static int utime_dir_fd(utime_t *ut, int dir_fd, char *path, int follow_symlinks) @@ -4588,9 +4586,7 @@ utime_dir_fd(utime_t *ut, int dir_fd, char *path, int follow_symlinks) #define FUTIMENSAT_DIR_FD_CONVERTER dir_fd_unavailable #endif -#define UTIME_HAVE_FD (defined(HAVE_FUTIMES) || defined(HAVE_FUTIMENS)) - -#if UTIME_HAVE_FD +#if defined(HAVE_FUTIMES) || defined(HAVE_FUTIMENS) static int utime_fd(utime_t *ut, int fd) @@ -4683,7 +4679,7 @@ os.utime dir_fd: dir_fd(requires='futimensat') = None follow_symlinks: bool=True -# "utime(path, times=None, *, ns=None, dir_fd=None, follow_symlinks=True)\n\ +# "utime(path, times=None, *[, ns], dir_fd=None, follow_symlinks=True)\n\ Set the access and modified time of path. @@ -4693,10 +4689,10 @@ On some platforms, path may also be specified as an open file descriptor. If times is not None, it must be a tuple (atime, mtime); atime and mtime should be expressed as float seconds since the epoch. -If ns is not None, it must be a tuple (atime_ns, mtime_ns); +If ns is specified, it must be a tuple (atime_ns, mtime_ns); atime_ns and mtime_ns should be expressed as integer nanoseconds since the epoch. -If both times and ns are None, utime uses the current time. +If times is None and ns is unspecified, utime uses the current time. Specifying tuples for both times and ns is an error. If dir_fd is not None, it should be a file descriptor open to a directory, @@ -4714,7 +4710,7 @@ dir_fd and follow_symlinks may not be available on your platform. static PyObject * os_utime_impl(PyModuleDef *module, path_t *path, PyObject *times, PyObject *ns, int dir_fd, int follow_symlinks) -/*[clinic end generated code: output=31f3434e560ba2f0 input=1f18c17d5941aa82]*/ +/*[clinic end generated code: output=31f3434e560ba2f0 input=081cdc54ca685385]*/ { #ifdef MS_WINDOWS HANDLE hFile; @@ -4835,13 +4831,13 @@ os_utime_impl(PyModuleDef *module, path_t *path, PyObject *times, else #endif -#if UTIME_HAVE_DIR_FD +#if defined(HAVE_FUTIMESAT) || defined(HAVE_UTIMENSAT) if ((dir_fd != DEFAULT_DIR_FD) || (!follow_symlinks)) result = utime_dir_fd(&utime, dir_fd, path->narrow, follow_symlinks); else #endif -#if UTIME_HAVE_FD +#if defined(HAVE_FUTIMES) || defined(HAVE_FUTIMENS) if (path->fd != -1) result = utime_fd(&utime, path->fd); else @@ -8239,10 +8235,10 @@ os_write_impl(PyModuleDef *module, int fd, Py_buffer *data) #ifdef HAVE_SENDFILE PyDoc_STRVAR(posix_sendfile__doc__, -"sendfile(out, in, offset, nbytes) -> byteswritten\n\ -sendfile(out, in, offset, nbytes, headers=None, trailers=None, flags=0)\n\ +"sendfile(out, in, offset, count) -> byteswritten\n\ +sendfile(out, in, offset, count[, headers][, trailers], flags=0)\n\ -> byteswritten\n\ -Copy nbytes bytes from file descriptor in to file descriptor out."); +Copy count bytes from file descriptor in to file descriptor out."); /* AC 3.5: don't bother converting, has optional group*/ static PyObject * @@ -8262,6 +8258,7 @@ posix_sendfile(PyObject *self, PyObject *args, PyObject *kwdict) off_t sbytes; struct sf_hdtr sf; int flags = 0; + /* Beware that "in" clashes with Python's own "in" operator keyword */ static char *keywords[] = {"out", "in", "offset", "count", "headers", "trailers", "flags", NULL}; @@ -8281,7 +8278,7 @@ posix_sendfile(PyObject *self, PyObject *args, PyObject *kwdict) if (headers != NULL) { if (!PySequence_Check(headers)) { PyErr_SetString(PyExc_TypeError, - "sendfile() headers must be a sequence or None"); + "sendfile() headers must be a sequence"); return NULL; } else { Py_ssize_t i = 0; /* Avoid uninitialized warning */ @@ -8298,7 +8295,7 @@ posix_sendfile(PyObject *self, PyObject *args, PyObject *kwdict) if (trailers != NULL) { if (!PySequence_Check(trailers)) { PyErr_SetString(PyExc_TypeError, - "sendfile() trailers must be a sequence or None"); + "sendfile() trailers must be a sequence"); return NULL; } else { Py_ssize_t i = 0; /* Avoid uninitialized warning */ diff --git a/Modules/pyexpat.c b/Modules/pyexpat.c index a43887e..9a6da73 100644 --- a/Modules/pyexpat.c +++ b/Modules/pyexpat.c @@ -1378,11 +1378,16 @@ static int xmlparse_setattro(xmlparseobject *self, PyObject *name, PyObject *v) { /* Set attribute 'name' to value 'v'. v==NULL means delete */ + if (!PyUnicode_Check(name)) { + PyErr_Format(PyExc_TypeError, + "attribute name must be string, not '%.200s'", + name->ob_type->tp_name); + return -1; + } if (v == NULL) { PyErr_SetString(PyExc_RuntimeError, "Cannot delete attribute"); return -1; } - assert(PyUnicode_Check(name)); if (PyUnicode_CompareWithASCIIString(name, "buffer_text") == 0) { int b = PyObject_IsTrue(v); if (b < 0) @@ -1435,17 +1440,18 @@ xmlparse_setattro(xmlparseobject *self, PyObject *name, PyObject *v) return -1; } - new_buffer_size=PyLong_AS_LONG(v); + new_buffer_size = PyLong_AsLong(v); + if (new_buffer_size <= 0) { + if (!PyErr_Occurred()) + PyErr_SetString(PyExc_ValueError, "buffer_size must be greater than zero"); + return -1; + } + /* trivial case -- no change */ if (new_buffer_size == self->buffer_size) { return 0; } - if (new_buffer_size <= 0) { - PyErr_SetString(PyExc_ValueError, "buffer_size must be greater than zero"); - return -1; - } - /* check maximum */ if (new_buffer_size > INT_MAX) { char errmsg[100]; diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c index ee24907..d9c70f8 100644 --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -5513,9 +5513,7 @@ socket_getaddrinfo(PyObject *self, PyObject *args, PyObject* kwargs) if (hobj == Py_None) { hptr = NULL; } else if (PyUnicode_Check(hobj)) { - _Py_IDENTIFIER(encode); - - idna = _PyObject_CallMethodId(hobj, &PyId_encode, "s", "idna"); + idna = PyUnicode_AsEncodedString(hobj, "idna", NULL); if (!idna) return NULL; assert(PyBytes_Check(idna)); diff --git a/Objects/odictobject.c b/Objects/odictobject.c index 7dbaa89..104d6c1 100644 --- a/Objects/odictobject.c +++ b/Objects/odictobject.c @@ -98,7 +98,6 @@ For removing nodes: Others: -* _odict_initialize(od) * _odict_find_node(od, key) * _odict_keys_equal(od1, od2) @@ -602,15 +601,6 @@ _odict_get_index(PyODictObject *od, PyObject *key) return _odict_get_index_hash(od, key, hash); } -static int -_odict_initialize(PyODictObject *od) -{ - od->od_state = 0; - _odict_FIRST(od) = NULL; - _odict_LAST(od) = NULL; - return _odict_resize((PyODictObject *)od); -} - /* Returns NULL if there was some error or the key was not found. */ static _ODictNode * _odict_find_node(PyODictObject *od, PyObject *key) @@ -744,7 +734,7 @@ _odict_pop_node(PyODictObject *od, _ODictNode *node, PyObject *key) /* If someone calls PyDict_DelItem() directly on an OrderedDict, we'll get all sorts of problems here. In PyODict_DelItem we make sure to call _odict_clear_node first. - + This matters in the case of colliding keys. Suppose we add 3 keys: [A, B, C], where the hash of C collides with A and the next possible index in the hash table is occupied by B. If we remove B then for C @@ -1739,14 +1729,28 @@ odict_init(PyObject *self, PyObject *args, PyObject *kwds) static PyObject * odict_new(PyTypeObject *type, PyObject *args, PyObject *kwds) { - PyObject *od = PyDict_Type.tp_new(type, args, kwds); - if (od != NULL) { - if (_odict_initialize((PyODictObject *)od) < 0) - return NULL; - ((PyODictObject *)od)->od_inst_dict = PyDict_New(); - ((PyODictObject *)od)->od_weakreflist = NULL; + PyObject *dict; + PyODictObject *od; + + dict = PyDict_New(); + if (dict == NULL) + return NULL; + + od = (PyODictObject *)PyDict_Type.tp_new(type, args, kwds); + if (od == NULL) { + Py_DECREF(dict); + return NULL; } - return od; + + od->od_inst_dict = dict; + /* type constructor fills the memory with zeros (see + PyType_GenericAlloc()), there is no need to set them to zero again */ + if (_odict_resize(od) < 0) { + Py_DECREF(od); + return NULL; + } + + return (PyObject*)od; } /* PyODict_Type */ diff --git a/PC/example_nt/example.c b/PC/example_nt/example.c deleted file mode 100644 index 669d11b..0000000 --- a/PC/example_nt/example.c +++ /dev/null @@ -1,32 +0,0 @@ -#include "Python.h" - -static PyObject * -ex_foo(PyObject *self, PyObject *args) -{ - printf("Hello, world\n"); - Py_INCREF(Py_None); - return Py_None; -} - -static PyMethodDef example_methods[] = { - {"foo", ex_foo, METH_VARARGS, "foo() doc string"}, - {NULL, NULL} -}; - -static struct PyModuleDef examplemodule = { - PyModuleDef_HEAD_INIT, - "example", - "example module doc string", - -1, - example_methods, - NULL, - NULL, - NULL, - NULL -}; - -PyMODINIT_FUNC -PyInit_example(void) -{ - return PyModule_Create(&examplemodule); -} diff --git a/PC/example_nt/example.sln b/PC/example_nt/example.sln deleted file mode 100644 index d8a3119..0000000 --- a/PC/example_nt/example.sln +++ /dev/null @@ -1,21 +0,0 @@ -Microsoft Visual Studio Solution File, Format Version 8.00
-Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "example", "example.vcproj", "{A0608D6F-84ED-44AE-A2A6-A3CC7F4A4030}"
- ProjectSection(ProjectDependencies) = postProject
- EndProjectSection
-EndProject
-Global
- GlobalSection(SolutionConfiguration) = preSolution
- Debug = Debug
- Release = Release
- EndGlobalSection
- GlobalSection(ProjectConfiguration) = postSolution
- {A0608D6F-84ED-44AE-A2A6-A3CC7F4A4030}.Debug.ActiveCfg = Debug|Win32
- {A0608D6F-84ED-44AE-A2A6-A3CC7F4A4030}.Debug.Build.0 = Debug|Win32
- {A0608D6F-84ED-44AE-A2A6-A3CC7F4A4030}.Release.ActiveCfg = Release|Win32
- {A0608D6F-84ED-44AE-A2A6-A3CC7F4A4030}.Release.Build.0 = Release|Win32
- EndGlobalSection
- GlobalSection(ExtensibilityGlobals) = postSolution
- EndGlobalSection
- GlobalSection(ExtensibilityAddIns) = postSolution
- EndGlobalSection
-EndGlobal
diff --git a/PC/example_nt/example.vcproj b/PC/example_nt/example.vcproj deleted file mode 100644 index d82f76e..0000000 --- a/PC/example_nt/example.vcproj +++ /dev/null @@ -1,189 +0,0 @@ -<?xml version="1.0" encoding="windows-1250"?>
-<VisualStudioProject
- ProjectType="Visual C++"
- Version="7.10"
- Name="example"
- SccProjectName=""
- SccLocalPath="">
- <Platforms>
- <Platform
- Name="Win32"/>
- </Platforms>
- <Configurations>
- <Configuration
- Name="Release|Win32"
- OutputDirectory=".\Release"
- IntermediateDirectory=".\Release"
- ConfigurationType="2"
- UseOfMFC="0"
- ATLMinimizesCRunTimeLibraryUsage="FALSE">
- <Tool
- Name="VCCLCompilerTool"
- Optimization="2"
- InlineFunctionExpansion="1"
- AdditionalIncludeDirectories="..\Include,..\PC"
- PreprocessorDefinitions="WIN32;NDEBUG;_WINDOWS"
- StringPooling="TRUE"
- RuntimeLibrary="2"
- EnableFunctionLevelLinking="TRUE"
- UsePrecompiledHeader="2"
- PrecompiledHeaderFile=".\Release/example.pch"
- AssemblerListingLocation=".\Release/"
- ObjectFile=".\Release/"
- ProgramDataBaseFileName=".\Release/"
- WarningLevel="3"
- SuppressStartupBanner="TRUE"
- CompileAs="0"/>
- <Tool
- Name="VCCustomBuildTool"/>
- <Tool
- Name="VCLinkerTool"
- AdditionalOptions="/export:initexample"
- AdditionalDependencies="odbc32.lib odbccp32.lib python35.lib"
- OutputFile=".\Release/example.pyd"
- LinkIncremental="1"
- SuppressStartupBanner="TRUE"
- AdditionalLibraryDirectories="..\PCbuild"
- ModuleDefinitionFile=""
- ProgramDatabaseFile=".\Release/example.pdb"
- SubSystem="2"
- ImportLibrary=".\Release/example.lib"
- TargetMachine="1"/>
- <Tool
- Name="VCMIDLTool"
- PreprocessorDefinitions="NDEBUG"
- MkTypLibCompatible="TRUE"
- SuppressStartupBanner="TRUE"
- TargetEnvironment="1"
- TypeLibraryName=".\Release/example.tlb"
- HeaderFileName=""/>
- <Tool
- Name="VCPostBuildEventTool"/>
- <Tool
- Name="VCPreBuildEventTool"/>
- <Tool
- Name="VCPreLinkEventTool"/>
- <Tool
- Name="VCResourceCompilerTool"
- PreprocessorDefinitions="NDEBUG"
- Culture="1033"/>
- <Tool
- Name="VCWebServiceProxyGeneratorTool"/>
- <Tool
- Name="VCXMLDataGeneratorTool"/>
- <Tool
- Name="VCWebDeploymentTool"/>
- <Tool
- Name="VCManagedWrapperGeneratorTool"/>
- <Tool
- Name="VCAuxiliaryManagedWrapperGeneratorTool"/>
- </Configuration>
- <Configuration
- Name="Debug|Win32"
- OutputDirectory=".\Debug"
- IntermediateDirectory=".\Debug"
- ConfigurationType="2"
- UseOfMFC="0"
- ATLMinimizesCRunTimeLibraryUsage="FALSE">
- <Tool
- Name="VCCLCompilerTool"
- Optimization="0"
- AdditionalIncludeDirectories="..\Include,..\PC"
- PreprocessorDefinitions="WIN32;_DEBUG;_WINDOWS"
- RuntimeLibrary="3"
- UsePrecompiledHeader="2"
- PrecompiledHeaderFile=".\Debug/example.pch"
- AssemblerListingLocation=".\Debug/"
- ObjectFile=".\Debug/"
- ProgramDataBaseFileName=".\Debug/"
- WarningLevel="3"
- SuppressStartupBanner="TRUE"
- DebugInformationFormat="4"
- CompileAs="0"/>
- <Tool
- Name="VCCustomBuildTool"/>
- <Tool
- Name="VCLinkerTool"
- AdditionalOptions="/export:initexample"
- AdditionalDependencies="odbc32.lib odbccp32.lib python35_d.lib"
- OutputFile=".\Debug/example_d.pyd"
- LinkIncremental="1"
- SuppressStartupBanner="TRUE"
- AdditionalLibraryDirectories="..\PCbuild"
- ModuleDefinitionFile=""
- GenerateDebugInformation="TRUE"
- ProgramDatabaseFile=".\Debug/example_d.pdb"
- SubSystem="2"
- ImportLibrary=".\Debug/example_d.lib"
- TargetMachine="1"/>
- <Tool
- Name="VCMIDLTool"
- PreprocessorDefinitions="_DEBUG"
- MkTypLibCompatible="TRUE"
- SuppressStartupBanner="TRUE"
- TargetEnvironment="1"
- TypeLibraryName=".\Debug/example.tlb"
- HeaderFileName=""/>
- <Tool
- Name="VCPostBuildEventTool"/>
- <Tool
- Name="VCPreBuildEventTool"/>
- <Tool
- Name="VCPreLinkEventTool"/>
- <Tool
- Name="VCResourceCompilerTool"
- PreprocessorDefinitions="_DEBUG"
- Culture="1033"/>
- <Tool
- Name="VCWebServiceProxyGeneratorTool"/>
- <Tool
- Name="VCXMLDataGeneratorTool"/>
- <Tool
- Name="VCWebDeploymentTool"/>
- <Tool
- Name="VCManagedWrapperGeneratorTool"/>
- <Tool
- Name="VCAuxiliaryManagedWrapperGeneratorTool"/>
- </Configuration>
- </Configurations>
- <References>
- </References>
- <Files>
- <Filter
- Name="Source Files"
- Filter="cpp;c;cxx;rc;def;r;odl;hpj;bat;for;f90">
- <File
- RelativePath="example.c">
- <FileConfiguration
- Name="Release|Win32">
- <Tool
- Name="VCCLCompilerTool"
- Optimization="2"
- AdditionalIncludeDirectories=""
- PreprocessorDefinitions="WIN32;NDEBUG;_WINDOWS;$(NoInherit)"/>
- </FileConfiguration>
- <FileConfiguration
- Name="Debug|Win32">
- <Tool
- Name="VCCLCompilerTool"
- Optimization="0"
- AdditionalIncludeDirectories=""
- PreprocessorDefinitions="WIN32;_DEBUG;_WINDOWS;$(NoInherit)"/>
- </FileConfiguration>
- </File>
- </Filter>
- <Filter
- Name="Header Files"
- Filter="h;hpp;hxx;hm;inl;fi;fd">
- </Filter>
- <Filter
- Name="Resource Files"
- Filter="ico;cur;bmp;dlg;rc2;rct;bin;cnt;rtf;gif;jpg;jpeg;jpe">
- </Filter>
- <File
- RelativePath="readme.txt">
- </File>
- </Files>
- <Globals>
- </Globals>
-</VisualStudioProject>
diff --git a/PC/example_nt/readme.txt b/PC/example_nt/readme.txt deleted file mode 100644 index d8efa74..0000000 --- a/PC/example_nt/readme.txt +++ /dev/null @@ -1,183 +0,0 @@ -Example Python extension for Windows NT -======================================= - -This directory contains everything needed (except for the Python -distribution!) to build a Python extension module using Microsoft VC++. -Notice that you need to use the same compiler version that was used to build -Python itself. - -The simplest way to build this example is to use the distutils script -'setup.py'. To do this, simply execute: - - % python setup.py install - -after everything builds and installs, you can test it: - - % python -c "import example; example.foo()" - Hello, world - -See setup.py for more details. alternatively, see below for instructions on -how to build inside the Visual Studio environment. - -Visual Studio Build Instructions -================================ - -These are instructions how to build an extension using Visual C++. The -instructions and project files have not been updated to the latest VC -version. In general, it is recommended you use the 'setup.py' instructions -above. - -It has been tested with VC++ 7.1 on Python 2.4. You can also use earlier -versions of VC to build Python extensions, but the sample VC project file -(example.dsw in this directory) is in VC 7.1 format. - -COPY THIS DIRECTORY! --------------------- -This "example_nt" directory is a subdirectory of the PC directory, in order -to keep all the PC-specific files under the same directory. However, the -example_nt directory can't actually be used from this location. You first -need to copy or move it up one level, so that example_nt is a direct -sibling of the PC\ and Include\ directories. Do all your work from within -this new location -- sorry, but you'll be sorry if you don't. - -OPEN THE PROJECT ----------------- -From VC 7.1, use the - File -> Open Solution... -dialog (*not* the "File -> Open..." dialog!). Navigate to and select the -file "example.sln", in the *copy* of the example_nt directory you made -above. -Click Open. - -BUILD THE EXAMPLE DLL ---------------------- -In order to check that everything is set up right, try building: - -1. Select a configuration. This step is optional. Do - Build -> Configuration Manager... -> Active Solution Configuration - and select either "Release" or "Debug". - If you skip this step, you'll use the Debug configuration by default. - -2. Build the DLL. Do - Build -> Build Solution - This creates all intermediate and result files in a subdirectory which - is called either Debug or Release, depending on which configuration you - picked in the preceding step. - -TESTING THE DEBUG-MODE DLL --------------------------- -Once the Debug build has succeeded, bring up a DOS box, and cd to -example_nt\Debug. You should now be able to repeat the following session -("C>" is the DOS prompt, ">>>" is the Python prompt) (note that various -debug output from Python may not match this screen dump exactly): - - C>..\..\PCbuild\python_d - Adding parser accelerators ... - Done. - Python 2.2c1+ (#28, Dec 14 2001, 18:06:39) [MSC 32 bit (Intel)] on win32 - Type "help", "copyright", "credits" or "license" for more information. - >>> import example - [7052 refs] - >>> example.foo() - Hello, world - [7052 refs] - >>> - -TESTING THE RELEASE-MODE DLL ----------------------------- -Once the Release build has succeeded, bring up a DOS box, and cd to -example_nt\Release. You should now be able to repeat the following session -("C>" is the DOS prompt, ">>>" is the Python prompt): - - C>..\..\PCbuild\python - Python 2.2c1+ (#28, Dec 14 2001, 18:06:04) [MSC 32 bit (Intel)] on win32 - Type "help", "copyright", "credits" or "license" for more information. - >>> import example - >>> example.foo() - Hello, world - >>> - -Congratulations! You've successfully built your first Python extension -module. - -CREATING YOUR OWN PROJECT -------------------------- -Choose a name ("spam" is always a winner :-) and create a directory for -it. Copy your C sources into it. Note that the module source file name -does not necessarily have to match the module name, but the "init" function -name should match the module name -- i.e. you can only import a module -"spam" if its init function is called "initspam()", and it should call -Py_InitModule with the string "spam" as its first argument (use the minimal -example.c in this directory as a guide). By convention, it lives in a file -called "spam.c" or "spammodule.c". The output file should be called -"spam.dll" or "spam.pyd" (the latter is supported to avoid confusion with a -system library "spam.dll" to which your module could be a Python interface) -in Release mode, or spam_d.dll or spam_d.pyd in Debug mode. - -Now your options are: - -1) Copy example.sln and example.vcproj, rename them to spam.*, and edit them -by hand. - -or - -2) Create a brand new project; instructions are below. - -In either case, copy example_nt\example.def to spam\spam.def, and edit the -new spam.def so its second line contains the string "initspam". If you -created a new project yourself, add the file spam.def to the project now. -(This is an annoying little file with only two lines. An alternative -approach is to forget about the .def file, and add the option -"/export:initspam" somewhere to the Link settings, by manually editing the -"Project -> Properties -> Linker -> Command Line -> Additional Options" -box). - -You are now all set to build your extension, unless it requires other -external libraries, include files, etc. See Python's Extending and -Embedding manual for instructions on how to write an extension. - - -CREATING A BRAND NEW PROJECT ----------------------------- -Use the - File -> New -> Project... -dialog to create a new Project Workspace. Select "Visual C++ Projects/Win32/ -Win32 Project", enter the name ("spam"), and make sure the "Location" is -set to parent of the spam directory you have created (which should be a direct -subdirectory of the Python build tree, a sibling of Include and PC). -In "Application Settings", select "DLL", and "Empty Project". Click OK. - -You should now create the file spam.def as instructed in the previous -section. Add the source files (including the .def file) to the project, -using "Project", "Add Existing Item". - -Now open the - Project -> spam properties... -dialog. (Impressive, isn't it? :-) You only need to change a few -settings. Make sure "All Configurations" is selected from the "Settings -for:" dropdown list. Select the "C/C++" tab. Choose the "General" -category in the popup menu at the top. Type the following text in the -entry box labeled "Addditional Include Directories:" - - ..\Include,..\PC - -Then, choose the "General" category in the "Linker" tab, and enter - ..\PCbuild -in the "Additional library Directories" box. - -Now you need to add some mode-specific settings (select "Accept" -when asked to confirm your changes): - -Select "Release" in the "Configuration" dropdown list. Click the -"Link" tab, choose the "Input" Category, and append "python24.lib" to the -list in the "Additional Dependencies" box. - -Select "Debug" in the "Settings for:" dropdown list, and append -"python24_d.lib" to the list in the Additional Dependencies" box. Then -click on the C/C++ tab, select "Code Generation", and select -"Multi-threaded Debug DLL" from the "Runtime library" dropdown list. - -Select "Release" again from the "Settings for:" dropdown list. -Select "Multi-threaded DLL" from the "Use run-time library:" dropdown list. - -That's all <wink>. diff --git a/PC/example_nt/setup.py b/PC/example_nt/setup.py deleted file mode 100644 index 0443bc7..0000000 --- a/PC/example_nt/setup.py +++ /dev/null @@ -1,22 +0,0 @@ -# This is an example of a distutils 'setup' script for the example_nt -# sample. This provides a simpler way of building your extension -# and means you can avoid keeping MSVC solution files etc in source-control. -# It also means it should magically build with all compilers supported by -# python. - -# USAGE: you probably want 'setup.py install' - but execute 'setup.py --help' -# for all the details. - -# NOTE: This is *not* a sample for distutils - it is just the smallest -# script that can build this. See distutils docs for more info. - -from distutils.core import setup, Extension - -example_mod = Extension('example', sources = ['example.c']) - - -setup(name = "example", - version = "1.0", - description = "A sample extension module", - ext_modules = [example_mod], -) diff --git a/PC/pyconfig.h b/PC/pyconfig.h index 324a130..d44173a 100644 --- a/PC/pyconfig.h +++ b/PC/pyconfig.h @@ -147,7 +147,11 @@ WIN32 is still required for the locale module. #define MS_WINI64 #define PYD_PLATFORM_TAG "win_ia64" #elif defined(_M_X64) || defined(_M_AMD64) +#if defined(__INTEL_COMPILER) +#define COMPILER ("[ICC v." _Py_STRINGIZE(__INTEL_COMPILER) " 64 bit (amd64) with MSC v." _Py_STRINGIZE(_MSC_VER) " CRT]") +#else #define COMPILER _Py_PASTE_VERSION("64 bit (AMD64)") +#endif /* __INTEL_COMPILER */ #define MS_WINX64 #define PYD_PLATFORM_TAG "win_amd64" #else @@ -194,7 +198,11 @@ typedef _W64 int ssize_t; #if defined(MS_WIN32) && !defined(MS_WIN64) #if defined(_M_IX86) +#if defined(__INTEL_COMPILER) +#define COMPILER ("[ICC v." _Py_STRINGIZE(__INTEL_COMPILER) " 32 bit (Intel) with MSC v." _Py_STRINGIZE(_MSC_VER) " CRT]") +#else #define COMPILER _Py_PASTE_VERSION("32 bit (Intel)") +#endif /* __INTEL_COMPILER */ #define PYD_PLATFORM_TAG "win32" #elif defined(_M_ARM) #define COMPILER _Py_PASTE_VERSION("32 bit (ARM)") diff --git a/PC/readme.txt b/PC/readme.txt index 8639dc3..0a96d26 100644 --- a/PC/readme.txt +++ b/PC/readme.txt @@ -71,8 +71,6 @@ getpathp.c Default sys.path calculations (for all PC platforms). dllbase_nt.txt A (manually maintained) list of base addresses for various DLLs, to avoid run-time relocation. -example_nt A subdirectory showing how to build an extension as a - DLL. Note for Windows 3.x and DOS users ================================== diff --git a/PCbuild/build.bat b/PCbuild/build.bat index 0fb01d7..2c41fb2 100644 --- a/PCbuild/build.bat +++ b/PCbuild/build.bat @@ -1,19 +1,46 @@ @echo off
-rem A batch program to build or rebuild a particular configuration,
-rem just for convenience.
-
-rem Arguments:
-rem -c Set the configuration (default: Release)
-rem -p Set the platform (x64 or Win32, default: Win32)
-rem -r Target Rebuild instead of Build
-rem -t Set the target manually (Build, Rebuild, Clean, or CleanAll)
-rem -d Set the configuration to Debug
-rem -e Pull in external libraries using get_externals.bat
-rem -m Enable parallel build (enabled by default)
-rem -M Disable parallel build
-rem -v Increased output messages
-rem -k Attempt to kill any running Pythons before building (usually unnecessary)
+goto Run
+:Usage
+echo.%~nx0 [flags and arguments] [quoted MSBuild options]
+echo.
+echo.Build CPython from the command line. Requires the appropriate
+echo.version(s) of Microsoft Visual Studio to be installed (see readme.txt).
+echo.Also requires Subversion (svn.exe) to be on PATH if the '-e' flag is
+echo.given.
+echo.
+echo.After the flags recognized by this script, up to 9 arguments to be passed
+echo.directly to MSBuild may be passed. If the argument contains an '=', the
+echo.entire argument must be quoted (e.g. `%~nx0 "/p:PlatformToolset=v100"`)
+echo.
+echo.Available flags:
+echo. -h Display this help message
+echo. -V Display version information for the current build
+echo. -r Target Rebuild instead of Build
+echo. -d Set the configuration to Debug
+echo. -e Build external libraries fetched by get_externals.bat
+echo. Extension modules that depend on external libraries will not attempt
+echo. to build if this flag is not present
+echo. -m Enable parallel build (enabled by default)
+echo. -M Disable parallel build
+echo. -v Increased output messages
+echo. -k Attempt to kill any running Pythons before building (usually done
+echo. automatically by the pythoncore project)
+echo.
+echo.Available flags to avoid building certain modules.
+echo.These flags have no effect if '-e' is not given:
+echo. --no-ssl Do not attempt to build _ssl
+echo. --no-tkinter Do not attempt to build Tkinter
+echo.
+echo.Available arguments:
+echo. -c Release ^| Debug ^| PGInstrument ^| PGUpdate
+echo. Set the configuration (default: Release)
+echo. -p x64 ^| Win32
+echo. Set the platform (default: Win32)
+echo. -t Build ^| Rebuild ^| Clean ^| CleanAll
+echo. Set the target manually
+exit /b 127
+:Run
setlocal
set platf=Win32
set vs_platf=x86
@@ -25,17 +52,29 @@ set verbose=/nologo /v:m set kill=
:CheckOpts
+if "%~1"=="-h" goto Usage
if "%~1"=="-c" (set conf=%2) & shift & shift & goto CheckOpts
if "%~1"=="-p" (set platf=%2) & shift & shift & goto CheckOpts
if "%~1"=="-r" (set target=Rebuild) & shift & goto CheckOpts
if "%~1"=="-t" (set target=%2) & shift & shift & goto CheckOpts
if "%~1"=="-d" (set conf=Debug) & shift & goto CheckOpts
-if "%~1"=="-e" call "%dir%get_externals.bat" & shift & goto CheckOpts
if "%~1"=="-m" (set parallel=/m) & shift & goto CheckOpts
if "%~1"=="-M" (set parallel=) & shift & goto CheckOpts
if "%~1"=="-v" (set verbose=/v:n) & shift & goto CheckOpts
if "%~1"=="-k" (set kill=true) & shift & goto CheckOpts
if "%~1"=="-V" shift & goto Version
+rem These use the actual property names used by MSBuild. We could just let
+rem them in through the environment, but we specify them on the command line
+rem anyway for visibility so set defaults after this
+if "%~1"=="-e" (set IncludeExternals=true) & shift & goto CheckOpts
+if "%~1"=="--no-ssl" (set IncludeSSL=false) & shift & goto CheckOpts
+if "%~1"=="--no-tkinter" (set IncludeTkinter=false) & shift & goto CheckOpts
+
+if "%IncludeExternals%"=="" set IncludeExternals=false
+if "%IncludeSSL%"=="" set IncludeSSL=true
+if "%IncludeTkinter%"=="" set IncludeTkinter=true
+
+if "%IncludeExternals%"=="true" call "%dir%get_externals.bat"
if "%platf%"=="x64" (set vs_platf=x86_amd64)
@@ -43,14 +82,18 @@ rem Setup the environment call "%dir%env.bat" %vs_platf% >nul
if "%kill%"=="true" (
- msbuild /v:m /nologo /target:KillPython "%pcbuild%\pythoncore.vcxproj" /p:Configuration=%conf% /p:Platform=%platf% /p:KillPython=true
+ msbuild /v:m /nologo /target:KillPython "%dir%\pythoncore.vcxproj" /p:Configuration=%conf% /p:Platform=%platf% /p:KillPython=true
)
rem Call on MSBuild to do the work, echo the command.
rem Passing %1-9 is not the preferred option, but argument parsing in
rem batch is, shall we say, "lackluster"
echo on
-msbuild "%dir%pcbuild.proj" /t:%target% %parallel% %verbose% /p:Configuration=%conf% /p:Platform=%platf% %1 %2 %3 %4 %5 %6 %7 %8 %9
+msbuild "%dir%pcbuild.proj" /t:%target% %parallel% %verbose%^
+ /p:Configuration=%conf% /p:Platform=%platf%^
+ /p:IncludeExternals=%IncludeExternals%^
+ /p:IncludeSSL=%IncludeSSL% /p:IncludeTkinter=%IncludeTkinter%^
+ %1 %2 %3 %4 %5 %6 %7 %8 %9
@goto :eof
diff --git a/PCbuild/get_externals.bat b/PCbuild/get_externals.bat index f9b740f..0b3c08b 100644 --- a/PCbuild/get_externals.bat +++ b/PCbuild/get_externals.bat @@ -51,16 +51,17 @@ if ERRORLEVEL 9009 ( echo.Fetching external libraries...
-for %%e in (
- bzip2-1.0.6
- nasm-2.11.06
- openssl-1.0.2d
- tcl-core-8.6.4.2
- tk-8.6.4.2
- tix-8.4.3.6
- sqlite-3.8.11.0
- xz-5.0.5
- ) do (
+set libraries=
+set libraries=%libraries% bzip2-1.0.6
+if NOT "%IncludeSSL%"=="false" set libraries=%libraries% nasm-2.11.06
+if NOT "%IncludeSSL%"=="false" set libraries=%libraries% openssl-1.0.2d
+set libraries=%libraries% sqlite-3.8.11.0
+if NOT "%IncludeTkinter%"=="false" set libraries=%libraries% tcl-core-8.6.4.2
+if NOT "%IncludeTkinter%"=="false" set libraries=%libraries% tk-8.6.4.2
+if NOT "%IncludeTkinter%"=="false" set libraries=%libraries% tix-8.4.3.6
+set libraries=%libraries% xz-5.0.5
+
+for %%e in (%libraries%) do (
if exist %%e (
echo.%%e already exists, skipping.
) else (
diff --git a/PCbuild/pcbuild.proj b/PCbuild/pcbuild.proj index 813831e..36621c9 100644 --- a/PCbuild/pcbuild.proj +++ b/PCbuild/pcbuild.proj @@ -5,8 +5,10 @@ <Platform Condition="'$(Platform)' == ''">Win32</Platform> <Configuration Condition="'$(Configuration)' == ''">Release</Configuration> <IncludeExtensions Condition="'$(IncludeExtensions)' == ''">true</IncludeExtensions> + <IncludeExternals Condition="'$(IncludeExternals)' == ''">true</IncludeExternals> <IncludeTests Condition="'$(IncludeTest)' == ''">true</IncludeTests> <IncludeSSL Condition="'$(IncludeSSL)' == ''">true</IncludeSSL> + <IncludeTkinter Condition="'$(IncludeTkinter)' == ''">true</IncludeTkinter> </PropertyGroup> <ItemDefinitionGroup> @@ -25,7 +27,7 @@ <!-- Parallel build is explicitly disabled for this project because it causes many conflicts between pythoncore and projects that depend - in pythoncore. Once the core DLL has been built, subsequent + on pythoncore. Once the core DLL has been built, subsequent projects will be built in parallel. --> <Projects Include="pythoncore.vcxproj"> @@ -40,10 +42,14 @@ <!-- _freeze_importlib --> <Projects Include="_freeze_importlib.vcxproj" /> <!-- Extension modules --> - <ExtensionModules Include="_bz2;_ctypes;_decimal;_elementtree;_lzma;_msi;_multiprocessing;_overlapped;_sqlite3;_tkinter;tix;pyexpat;select;unicodedata;winsound" /> + <ExtensionModules Include="_ctypes;_decimal;_elementtree;_msi;_multiprocessing;_overlapped;pyexpat;select;unicodedata;winsound" /> + <!-- Extension modules that require external sources --> + <ExternalModules Include="_bz2;_lzma;_sqlite3" /> <!-- _ssl will build _socket as well, which may cause conflicts in parallel builds --> - <ExtensionModules Include="_socket" Condition="!$(IncludeSSL)" /> - <ExtensionModules Include="_ssl;_hashlib" Condition="$(IncludeSSL)" /> + <ExtensionModules Include="_socket" Condition="!$(IncludeSSL) or !$(IncludeExternals)" /> + <ExternalModules Include="_ssl;_hashlib" Condition="$(IncludeSSL)" /> + <ExternalModules Include="_tkinter;tix" Condition="$(IncludeTkinter)" /> + <ExtensionModules Include="@(ExternalModules->'%(Identity)')" Condition="$(IncludeExternals)" /> <Projects Include="@(ExtensionModules->'%(Identity).vcxproj')" Condition="$(IncludeExtensions)" /> <!-- Test modules --> <TestModules Include="_ctypes_test;_testbuffer;_testcapi;_testembed;_testimportmultiple;_testmultiphase" /> diff --git a/PCbuild/readme.txt b/PCbuild/readme.txt index 68cdb0f..09a996f 100644 --- a/PCbuild/readme.txt +++ b/PCbuild/readme.txt @@ -41,8 +41,7 @@ The solution currently supports two platforms. The Win32 platform is used to build standard x86-compatible 32-bit binaries, output into the
win32 sub-directory. The x64 platform is used for building 64-bit AMD64
(aka x86_64 or EM64T) binaries, output into the amd64 sub-directory.
-The Itanium (IA-64) platform is no longer supported. See the "Building
-for AMD64" section below for more information about 64-bit builds.
+The Itanium (IA-64) platform is no longer supported.
Four configuration options are supported by the solution:
Debug
@@ -76,29 +75,7 @@ officially supported. By default, build.bat will build Python in Release configuration for
the 32-bit Win32 platform. It accepts several arguments to change
-this behavior:
-
- -c <configuration> Set the configuration (see above)
- -d Shortcut for "-c Debug"
- -p <platform> Set the platform to build for ("Win32" or "x64")
- -r Rebuild instead of just building
- -t <target> Set the target (Build, Rebuild, Clean or CleanAll)
- -e Use get_externals.bat to fetch external sources
- -M Don't build in parallel
- -v Increased output messages
-
-Up to 9 MSBuild switches can also be passed, though they must be passed
-after specifying any of the above switches. For example, use:
-
- build.bat -e -d /fl
-
-to do a debug build with externals fetched as needed and write detailed
-build logs to a file. If the MSBuild switch requires an equal sign
-("="), the entire switch must be quoted:
-
- build.bat -e -d "/p:ExternalsDir=P:\cpython-externals"
-
-There may also be other situations where quotes are necessary.
+this behavior, try `build.bat -h` to learn more.
C Runtime
@@ -129,8 +106,6 @@ pythoncore .dll and .lib
python
.exe
-make_buildinfo, make_versioninfo
- helpers to provide necessary information to the build process
These sub-projects provide extra executables that are useful for running
CPython in different ways:
@@ -152,9 +127,6 @@ categories: _freeze_importlib
_freeze_importlib.exe, used to regenerate Python\importlib.h after
changes have been made to Lib\importlib\_bootstrap.py
-bdist_wininst
- ..\Lib\distutils\command\wininst-14.0[-amd64].exe, the base
- executable used by the distutils bdist_wininst command
python3dll
python3.dll, the PEP 384 Stable ABI dll
xxlimited
@@ -274,14 +246,8 @@ as the values of certain properties in order for the build solution to find them. This is an advanced topic and not necessarily fully
supported.
-
-Building for AMD64
-------------------
-
-The build process for AMD64 / x64 is very similar to standard builds,
-you just have to set x64 as platform. In addition, the HOST_PYTHON
-environment variable must point to a Python interpreter (at least 2.4),
-to support cross-compilation from Win32.
+The get_externals.bat script is called automatically by build.bat when
+you pass the '-e' option to it.
Profile Guided Optimization
@@ -298,7 +264,7 @@ It creates the PGI files, runs the unit test suite or PyBench with the PGI python, and finally creates the optimized files.
See
- http://msdn.microsoft.com/en-us/library/e7k32f4k(VS.100).aspx
+ http://msdn.microsoft.com/en-us/library/e7k32f4k(VS.140).aspx
for more on this topic.
@@ -332,11 +298,3 @@ project, with some projects overriding certain specific values. The GUI doesn't always reflect the correct settings and may confuse the user
with false information, especially for settings that automatically adapt
for diffirent configurations.
-
-
-Your Own Extension DLLs
------------------------
-
-If you want to create your own extension module DLL (.pyd), there's an
-example with easy-to-follow instructions in ..\PC\example\; read the
-file readme.txt there first.
diff --git a/PCbuild/rt.bat b/PCbuild/rt.bat index 039c810..2d93b80 100644 --- a/PCbuild/rt.bat +++ b/PCbuild/rt.bat @@ -32,15 +32,17 @@ set prefix=%pcbuild%win32\ set suffix=
set qmode=
set dashO=
+set regrtestargs=
:CheckOpts
if "%1"=="-O" (set dashO=-O) & shift & goto CheckOpts
if "%1"=="-q" (set qmode=yes) & shift & goto CheckOpts
if "%1"=="-d" (set suffix=_d) & shift & goto CheckOpts
if "%1"=="-x64" (set prefix=%pcbuild%amd64\) & shift & goto CheckOpts
+if NOT "%1"=="" (set regrtestargs=%regrtestargs% %1) & shift & goto CheckOpts
set exe=%prefix%python%suffix%.exe
-set cmd="%exe%" %dashO% -Wd -E -bb "%pcbuild%..\lib\test\regrtest.py" %1 %2 %3 %4 %5 %6 %7 %8 %9
+set cmd="%exe%" %dashO% -Wd -E -bb "%pcbuild%..\lib\test\regrtest.py" %regrtestargs%
if defined qmode goto Qmode
echo Deleting .pyc/.pyo files ...
diff --git a/Python/codecs.c b/Python/codecs.c index 596bd80..d90bf73 100644 --- a/Python/codecs.c +++ b/Python/codecs.c @@ -966,7 +966,6 @@ PyObject *PyCodec_BackslashReplaceErrors(PyObject *exc) } static _PyUnicode_Name_CAPI *ucnhash_CAPI = NULL; -static int ucnhash_initialized = 0; PyObject *PyCodec_NameReplaceErrors(PyObject *exc) { @@ -988,17 +987,17 @@ PyObject *PyCodec_NameReplaceErrors(PyObject *exc) return NULL; if (!(object = PyUnicodeEncodeError_GetObject(exc))) return NULL; - if (!ucnhash_initialized) { + if (!ucnhash_CAPI) { /* load the unicode data module */ ucnhash_CAPI = (_PyUnicode_Name_CAPI *)PyCapsule_Import( PyUnicodeData_CAPSULE_NAME, 1); - ucnhash_initialized = 1; + if (!ucnhash_CAPI) + return NULL; } for (i = start, ressize = 0; i < end; ++i) { /* object is guaranteed to be "ready" */ c = PyUnicode_READ_CHAR(object, i); - if (ucnhash_CAPI && - ucnhash_CAPI->getname(NULL, c, buffer, sizeof(buffer), 1)) { + if (ucnhash_CAPI->getname(NULL, c, buffer, sizeof(buffer), 1)) { replsize = 1+1+1+(int)strlen(buffer)+1; } else if (c >= 0x10000) { @@ -1021,8 +1020,7 @@ PyObject *PyCodec_NameReplaceErrors(PyObject *exc) i < end; ++i) { c = PyUnicode_READ_CHAR(object, i); *outp++ = '\\'; - if (ucnhash_CAPI && - ucnhash_CAPI->getname(NULL, c, buffer, sizeof(buffer), 1)) { + if (ucnhash_CAPI->getname(NULL, c, buffer, sizeof(buffer), 1)) { *outp++ = 'N'; *outp++ = '{'; strcpy((char *)outp, buffer); diff --git a/Python/compile.c b/Python/compile.c index cfeab0f..97bb12e 100644 --- a/Python/compile.c +++ b/Python/compile.c @@ -985,7 +985,7 @@ PyCompile_OpcodeStackEffect(int opcode, int oparg) case BUILD_MAP_UNPACK_WITH_CALL: return 1 - (oparg & 0xFF); case BUILD_MAP: - return 1; + return 1 - 2*oparg; case LOAD_ATTR: return 0; case COMPARE_OP: diff --git a/Python/importlib_external.h b/Python/importlib_external.h index aea434c..48d6be5 100644 --- a/Python/importlib_external.h +++ b/Python/importlib_external.h @@ -1395,7 +1395,7 @@ const unsigned char _Py_M__importlib_external[] = { 32,83,111,117,114,99,101,76,111,97,100,101,114,32,117,115, 105,110,103,32,116,104,101,32,102,105,108,101,32,115,121,115, 116,101,109,46,99,2,0,0,0,0,0,0,0,3,0,0, - 0,5,0,0,0,67,0,0,0,115,34,0,0,0,116,0, + 0,4,0,0,0,67,0,0,0,115,34,0,0,0,116,0, 0,124,1,0,131,1,0,125,2,0,100,1,0,124,2,0, 106,1,0,100,2,0,124,2,0,106,2,0,105,2,0,83, 41,3,122,33,82,101,116,117,114,110,32,116,104,101,32,109, diff --git a/Python/pylifecycle.c b/Python/pylifecycle.c index a17adf7..df66fa0 100644 --- a/Python/pylifecycle.c +++ b/Python/pylifecycle.c @@ -1,4 +1,3 @@ - /* Python interpreter top-level routines, including init/exit */ #include "Python.h" @@ -963,6 +962,23 @@ initsite(void) } } +/* Check if a file descriptor is valid or not. + Return 0 if the file descriptor is invalid, return non-zero otherwise. */ +static int +is_valid_fd(int fd) +{ + int fd2; + if (fd < 0 || !_PyVerify_fd(fd)) + return 0; + _Py_BEGIN_SUPPRESS_IPH + fd2 = dup(fd); + if (fd2 >= 0) + close(fd2); + _Py_END_SUPPRESS_IPH + return fd2 >= 0; +} + +/* returns Py_None if the fd is not valid */ static PyObject* create_stdio(PyObject* io, int fd, int write_mode, char* name, @@ -978,6 +994,9 @@ create_stdio(PyObject* io, _Py_IDENTIFIER(TextIOWrapper); _Py_IDENTIFIER(mode); + if (!is_valid_fd(fd)) + Py_RETURN_NONE; + /* stdin is always opened in buffered mode, first because it shouldn't make a difference in common use cases, second because TextIOWrapper depends on the presence of a read1() method which only exists on @@ -1059,21 +1078,15 @@ error: Py_XDECREF(stream); Py_XDECREF(text); Py_XDECREF(raw); - return NULL; -} -static int -is_valid_fd(int fd) -{ - int dummy_fd; - if (fd < 0 || !_PyVerify_fd(fd)) - return 0; - _Py_BEGIN_SUPPRESS_IPH - dummy_fd = dup(fd); - if (dummy_fd >= 0) - close(dummy_fd); - _Py_END_SUPPRESS_IPH - return dummy_fd >= 0; + if (PyErr_ExceptionMatches(PyExc_OSError) && !is_valid_fd(fd)) { + /* Issue #24891: the file descriptor was closed after the first + is_valid_fd() check was called. Ignore the OSError and set the + stream to None. */ + PyErr_Clear(); + Py_RETURN_NONE; + } + return NULL; } /* Initialize sys.stdin, stdout, stderr and builtins.open */ @@ -1158,30 +1171,18 @@ initstdio(void) * and fileno() may point to an invalid file descriptor. For example * GUI apps don't have valid standard streams by default. */ - if (!is_valid_fd(fd)) { - std = Py_None; - Py_INCREF(std); - } - else { - std = create_stdio(iomod, fd, 0, "<stdin>", encoding, errors); - if (std == NULL) - goto error; - } /* if (fd < 0) */ + std = create_stdio(iomod, fd, 0, "<stdin>", encoding, errors); + if (std == NULL) + goto error; PySys_SetObject("__stdin__", std); _PySys_SetObjectId(&PyId_stdin, std); Py_DECREF(std); /* Set sys.stdout */ fd = fileno(stdout); - if (!is_valid_fd(fd)) { - std = Py_None; - Py_INCREF(std); - } - else { - std = create_stdio(iomod, fd, 1, "<stdout>", encoding, errors); - if (std == NULL) - goto error; - } /* if (fd < 0) */ + std = create_stdio(iomod, fd, 1, "<stdout>", encoding, errors); + if (std == NULL) + goto error; PySys_SetObject("__stdout__", std); _PySys_SetObjectId(&PyId_stdout, std); Py_DECREF(std); @@ -1189,15 +1190,9 @@ initstdio(void) #if 1 /* Disable this if you have trouble debugging bootstrap stuff */ /* Set sys.stderr, replaces the preliminary stderr */ fd = fileno(stderr); - if (!is_valid_fd(fd)) { - std = Py_None; - Py_INCREF(std); - } - else { - std = create_stdio(iomod, fd, 1, "<stderr>", encoding, "backslashreplace"); - if (std == NULL) - goto error; - } /* if (fd < 0) */ + std = create_stdio(iomod, fd, 1, "<stderr>", encoding, "backslashreplace"); + if (std == NULL) + goto error; /* Same as hack above, pre-import stderr's codec to avoid recursion when import.c tries to write to stderr in verbose mode. */ diff --git a/Python/pytime.c b/Python/pytime.c index 02a9374..77db204 100644 --- a/Python/pytime.c +++ b/Python/pytime.c @@ -531,12 +531,8 @@ _PyTime_GetSystemClockWithInfo(_PyTime_t *t, _Py_clock_info_t *info) static int -pymonotonic_new(_PyTime_t *tp, _Py_clock_info_t *info, int raise) +pymonotonic(_PyTime_t *tp, _Py_clock_info_t *info, int raise) { -#ifdef Py_DEBUG - static int last_set = 0; - static _PyTime_t last = 0; -#endif #if defined(MS_WINDOWS) ULONGLONG result; @@ -628,12 +624,6 @@ pymonotonic_new(_PyTime_t *tp, _Py_clock_info_t *info, int raise) if (_PyTime_FromTimespec(tp, &ts, raise) < 0) return -1; #endif -#ifdef Py_DEBUG - /* monotonic clock cannot go backward */ - assert(!last_set || last <= *tp); - last = *tp; - last_set = 1; -#endif return 0; } @@ -641,7 +631,7 @@ _PyTime_t _PyTime_GetMonotonicClock(void) { _PyTime_t t; - if (pymonotonic_new(&t, NULL, 0) < 0) { + if (pymonotonic(&t, NULL, 0) < 0) { /* should not happen, _PyTime_Init() checked that monotonic clock at startup */ assert(0); @@ -655,7 +645,7 @@ _PyTime_GetMonotonicClock(void) int _PyTime_GetMonotonicClockWithInfo(_PyTime_t *tp, _Py_clock_info_t *info) { - return pymonotonic_new(tp, info, 1); + return pymonotonic(tp, info, 1); } int diff --git a/Tools/buildbot/test.bat b/Tools/buildbot/test.bat index 154dfa5..d1f5f52 100644 --- a/Tools/buildbot/test.bat +++ b/Tools/buildbot/test.bat @@ -1,15 +1,19 @@ -@rem Used by the buildbot "test" step.
-@setlocal
+@echo off
+rem Used by the buildbot "test" step.
+setlocal
-@set here=%~dp0
-@set rt_opts=-q -d
+set here=%~dp0
+set rt_opts=-q -d
+set regrtest_args=
:CheckOpts
-@if '%1'=='-x64' (set rt_opts=%rt_opts% %1) & shift & goto CheckOpts
-@if '%1'=='-d' (set rt_opts=%rt_opts% %1) & shift & goto CheckOpts
-@if '%1'=='-O' (set rt_opts=%rt_opts% %1) & shift & goto CheckOpts
-@if '%1'=='-q' (set rt_opts=%rt_opts% %1) & shift & goto CheckOpts
-@if '%1'=='+d' (set rt_opts=%rt_opts:-d=%) & shift & goto CheckOpts
-@if '%1'=='+q' (set rt_opts=%rt_opts:-q=%) & shift & goto CheckOpts
+if "%1"=="-x64" (set rt_opts=%rt_opts% %1) & shift & goto CheckOpts
+if "%1"=="-d" (set rt_opts=%rt_opts% %1) & shift & goto CheckOpts
+if "%1"=="-O" (set rt_opts=%rt_opts% %1) & shift & goto CheckOpts
+if "%1"=="-q" (set rt_opts=%rt_opts% %1) & shift & goto CheckOpts
+if "%1"=="+d" (set rt_opts=%rt_opts:-d=%) & shift & goto CheckOpts
+if "%1"=="+q" (set rt_opts=%rt_opts:-q=%) & shift & goto CheckOpts
+if NOT "%1"=="" (set regrtest_args=%regrtest_args% %1) & shift & goto CheckOpts
-call "%here%..\..\PCbuild\rt.bat" %rt_opts% -uall -rwW -n --timeout=3600 %1 %2 %3 %4 %5 %6 %7 %8 %9
+echo on
+call "%here%..\..\PCbuild\rt.bat" %rt_opts% -uall -rwW -n --timeout=3600 %regrtest_args%
diff --git a/Tools/msi/build.bat b/Tools/msi/build.bat index 008c937..5f53a1b 100644 --- a/Tools/msi/build.bat +++ b/Tools/msi/build.bat @@ -7,6 +7,7 @@ set BUILDX86= set BUILDX64=
set BUILDDOC=
set BUILDPX=
+set BUILDPACK=
:CheckOpts
if "%~1" EQU "-h" goto Help
@@ -14,6 +15,7 @@ if "%~1" EQU "-x86" (set BUILDX86=1) && shift && goto CheckOpts if "%~1" EQU "-x64" (set BUILDX64=1) && shift && goto CheckOpts
if "%~1" EQU "--doc" (set BUILDDOC=1) && shift && goto CheckOpts
if "%~1" EQU "--test-marker" (set BUILDPX=1) && shift && goto CheckOpts
+if "%~1" EQU "--pack" (set BUILDPACK=1) && shift && goto CheckOpts
if not defined BUILDX86 if not defined BUILDX64 (set BUILDX86=1) && (set BUILDX64=1)
@@ -41,6 +43,9 @@ set BUILD_CMD="%D%bundle\snapshot.wixproj" if defined BUILDPX (
set BUILD_CMD=%BUILD_CMD% /p:UseTestMarker=true
)
+if defined BUILDPACK (
+ set BUILD_CMD=%BUILD_CMD% /p:Pack=true
+)
if defined BUILDX86 (
"%PCBUILD%win32\python.exe" "%D%get_wix.py"
@@ -56,9 +61,10 @@ if defined BUILDX64 ( exit /B 0
:Help
-echo build.bat [-x86] [-x64] [--doc] [-h] [--test-marker]
+echo build.bat [-x86] [-x64] [--doc] [-h] [--test-marker] [--pack]
echo.
echo -x86 Build x86 installers
echo -x64 Build x64 installers
echo --doc Build CHM documentation
echo --test-marker Build installers with 'x' markers
+echo --pack Embed core MSIs into installer
diff --git a/Tools/msi/bundle/snapshot.wixproj b/Tools/msi/bundle/snapshot.wixproj index 8fe95a8..cc45043 100644 --- a/Tools/msi/bundle/snapshot.wixproj +++ b/Tools/msi/bundle/snapshot.wixproj @@ -9,9 +9,14 @@ <Import Project="..\msi.props" /> <PropertyGroup> + <DefineConstants Condition="'$(Pack)' != 'true'"> + $(DefineConstants);CompressMSI=no; + </DefineConstants> + <DefineConstants Condition="'$(Pack)' == 'true'"> + $(DefineConstants);CompressMSI=yes; + </DefineConstants> <DefineConstants> $(DefineConstants); - CompressMSI=no; CompressPDB=no; CompressMSI_D=no; </DefineConstants> diff --git a/Tools/msi/core/core_d.wxs b/Tools/msi/core/core_d.wxs index bb35f99..07e0397 100644 --- a/Tools/msi/core/core_d.wxs +++ b/Tools/msi/core/core_d.wxs @@ -1,6 +1,6 @@ <?xml version="1.0" encoding="UTF-8"?> <Wix xmlns="http://schemas.microsoft.com/wix/2006/wi"> - <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> + <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title_d)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" /> <MediaTemplate EmbedCab="yes" CompressionLevel="high" /> diff --git a/Tools/msi/core/core_pdb.wxs b/Tools/msi/core/core_pdb.wxs index ba72d95..c2c3178 100644 --- a/Tools/msi/core/core_pdb.wxs +++ b/Tools/msi/core/core_pdb.wxs @@ -1,6 +1,6 @@ <?xml version="1.0" encoding="UTF-8"?> <Wix xmlns="http://schemas.microsoft.com/wix/2006/wi"> - <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> + <Product Id="*" Language="!(loc.LCID)" Name="!(loc.TitlePdb)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" /> <MediaTemplate EmbedCab="yes" CompressionLevel="high" /> diff --git a/Tools/msi/dev/dev_d.wxs b/Tools/msi/dev/dev_d.wxs index c3cb2ea..c467aac 100644 --- a/Tools/msi/dev/dev_d.wxs +++ b/Tools/msi/dev/dev_d.wxs @@ -1,6 +1,6 @@ <?xml version="1.0" encoding="UTF-8"?> <Wix xmlns="http://schemas.microsoft.com/wix/2006/wi"> - <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> + <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title_d)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" /> <MediaTemplate EmbedCab="yes" CompressionLevel="high" /> diff --git a/Tools/msi/exe/exe_d.wxs b/Tools/msi/exe/exe_d.wxs index abcb012..eedb6bb 100644 --- a/Tools/msi/exe/exe_d.wxs +++ b/Tools/msi/exe/exe_d.wxs @@ -1,6 +1,6 @@ <?xml version="1.0" encoding="UTF-8"?> <Wix xmlns="http://schemas.microsoft.com/wix/2006/wi"> - <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> + <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title_d)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" /> <MediaTemplate EmbedCab="yes" CompressionLevel="high" /> diff --git a/Tools/msi/exe/exe_pdb.wxs b/Tools/msi/exe/exe_pdb.wxs index 5129ec0..f25094f 100644 --- a/Tools/msi/exe/exe_pdb.wxs +++ b/Tools/msi/exe/exe_pdb.wxs @@ -1,6 +1,6 @@ <?xml version="1.0" encoding="UTF-8"?> <Wix xmlns="http://schemas.microsoft.com/wix/2006/wi"> - <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> + <Product Id="*" Language="!(loc.LCID)" Name="!(loc.TitlePdb)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" /> <MediaTemplate EmbedCab="yes" CompressionLevel="high" /> diff --git a/Tools/msi/lib/lib_d.wxs b/Tools/msi/lib/lib_d.wxs index 5a5cf70..8a8a530 100644 --- a/Tools/msi/lib/lib_d.wxs +++ b/Tools/msi/lib/lib_d.wxs @@ -1,6 +1,6 @@ <?xml version="1.0" encoding="UTF-8"?> <Wix xmlns="http://schemas.microsoft.com/wix/2006/wi"> - <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> + <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title_d)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" /> <MediaTemplate EmbedCab="yes" CompressionLevel="high" /> diff --git a/Tools/msi/lib/lib_pdb.wxs b/Tools/msi/lib/lib_pdb.wxs index a2be0c9..8839e8a 100644 --- a/Tools/msi/lib/lib_pdb.wxs +++ b/Tools/msi/lib/lib_pdb.wxs @@ -1,6 +1,6 @@ <?xml version="1.0" encoding="UTF-8"?> <Wix xmlns="http://schemas.microsoft.com/wix/2006/wi"> - <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> + <Product Id="*" Language="!(loc.LCID)" Name="!(loc.TitlePdb)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" /> <MediaTemplate EmbedCab="yes" CompressionLevel="high" /> diff --git a/Tools/msi/make_zip.py b/Tools/msi/make_zip.py index 09ee491..c256008 100644 --- a/Tools/msi/make_zip.py +++ b/Tools/msi/make_zip.py @@ -3,6 +3,7 @@ import py_compile import re import sys import shutil +import stat import os import tempfile @@ -101,11 +102,16 @@ def copy_to_layout(target, rel_sources): else: for s, rel in rel_sources: + dest = target / rel try: - (target / rel).parent.mkdir(parents=True) + dest.parent.mkdir(parents=True) except FileExistsError: pass - shutil.copy(str(s), str(target / rel)) + if dest.is_file(): + dest.chmod(stat.S_IWRITE) + shutil.copy(str(s), str(dest)) + if dest.is_file(): + dest.chmod(stat.S_IWRITE) count += 1 return count diff --git a/Tools/msi/tcltk/tcltk_d.wxs b/Tools/msi/tcltk/tcltk_d.wxs index 7f5048f..01d0d24 100644 --- a/Tools/msi/tcltk/tcltk_d.wxs +++ b/Tools/msi/tcltk/tcltk_d.wxs @@ -1,6 +1,6 @@ <?xml version="1.0" encoding="UTF-8"?> <Wix xmlns="http://schemas.microsoft.com/wix/2006/wi"> - <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> + <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title_d)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" /> <MediaTemplate EmbedCab="yes" CompressionLevel="high" /> diff --git a/Tools/msi/tcltk/tcltk_pdb.wxs b/Tools/msi/tcltk/tcltk_pdb.wxs index 75c62bb..04454f3 100644 --- a/Tools/msi/tcltk/tcltk_pdb.wxs +++ b/Tools/msi/tcltk/tcltk_pdb.wxs @@ -1,6 +1,6 @@ <?xml version="1.0" encoding="UTF-8"?> <Wix xmlns="http://schemas.microsoft.com/wix/2006/wi"> - <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> + <Product Id="*" Language="!(loc.LCID)" Name="!(loc.TitlePdb)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" /> <MediaTemplate EmbedCab="yes" CompressionLevel="high" /> diff --git a/Tools/msi/test/test_d.wxs b/Tools/msi/test/test_d.wxs index a25afdd..a954876 100644 --- a/Tools/msi/test/test_d.wxs +++ b/Tools/msi/test/test_d.wxs @@ -1,6 +1,6 @@ <?xml version="1.0" encoding="UTF-8"?> <Wix xmlns="http://schemas.microsoft.com/wix/2006/wi"> - <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> + <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title_d)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" /> <MediaTemplate EmbedCab="yes" CompressionLevel="high" /> diff --git a/Tools/msi/test/test_pdb.wxs b/Tools/msi/test/test_pdb.wxs index 1510a6f..de634a3 100644 --- a/Tools/msi/test/test_pdb.wxs +++ b/Tools/msi/test/test_pdb.wxs @@ -1,6 +1,6 @@ <?xml version="1.0" encoding="UTF-8"?> <Wix xmlns="http://schemas.microsoft.com/wix/2006/wi"> - <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> + <Product Id="*" Language="!(loc.LCID)" Name="!(loc.TitlePdb)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" /> <MediaTemplate EmbedCab="yes" CompressionLevel="high" /> |