diff options
114 files changed, 1739 insertions, 1038 deletions
diff --git a/Doc/c-api/buffer.rst b/Doc/c-api/buffer.rst index 6cb474e..46c19d3 100644 --- a/Doc/c-api/buffer.rst +++ b/Doc/c-api/buffer.rst @@ -133,15 +133,15 @@ a buffer, see :c:func:`PyObject_GetBuffer`. called on non-NULL :c:member:`~Py_buffer.format` values. Important exception: If a consumer requests a buffer without the - :c:macro:`PyBUF_FORMAT` flag, :c:member:`~Py_Buffer.format` will + :c:macro:`PyBUF_FORMAT` flag, :c:member:`~Py_buffer.format` will be set to *NULL*, but :c:member:`~Py_buffer.itemsize` still has the value for the original format. - If :c:member:`~Py_Buffer.shape` is present, the equality + If :c:member:`~Py_buffer.shape` is present, the equality ``product(shape) * itemsize == len`` still holds and the consumer can use :c:member:`~Py_buffer.itemsize` to navigate the buffer. - If :c:member:`~Py_Buffer.shape` is *NULL* as a result of a :c:macro:`PyBUF_SIMPLE` + If :c:member:`~Py_buffer.shape` is *NULL* as a result of a :c:macro:`PyBUF_SIMPLE` or a :c:macro:`PyBUF_WRITABLE` request, the consumer must disregard :c:member:`~Py_buffer.itemsize` and assume ``itemsize == 1``. @@ -156,7 +156,7 @@ a buffer, see :c:func:`PyObject_GetBuffer`. .. c:member:: int ndim The number of dimensions the memory represents as an n-dimensional array. - If it is 0, :c:member:`~Py_Buffer.buf` points to a single item representing + If it is 0, :c:member:`~Py_buffer.buf` points to a single item representing a scalar. In this case, :c:member:`~Py_buffer.shape`, :c:member:`~Py_buffer.strides` and :c:member:`~Py_buffer.suboffsets` MUST be *NULL*. diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst index ed83e82..b5113aa 100644 --- a/Doc/c-api/typeobj.rst +++ b/Doc/c-api/typeobj.rst @@ -94,7 +94,7 @@ type objects) *must* have the :attr:`ob_size` field. This field is not inherited by subtypes. -.. c:member:: char* PyTypeObject.tp_name +.. c:member:: const char* PyTypeObject.tp_name Pointer to a NUL-terminated string containing the name of the type. For types that are accessible as module globals, the string should be the full module @@ -372,7 +372,7 @@ type objects) *must* have the :attr:`ob_size` field. inherited individually. -.. c:member:: long PyTypeObject.tp_flags +.. c:member:: unsigned long PyTypeObject.tp_flags This field is a bit mask of various flags. Some flags indicate variant semantics for certain situations; others are used to indicate that certain @@ -472,7 +472,7 @@ type objects) *must* have the :attr:`ob_size` field. .. versionadded:: 3.4 -.. c:member:: char* PyTypeObject.tp_doc +.. c:member:: const char* PyTypeObject.tp_doc An optional pointer to a NUL-terminated C string giving the docstring for this type object. This is exposed as the :attr:`__doc__` attribute on the type and @@ -619,7 +619,7 @@ type objects) *must* have the :attr:`ob_size` field. +----------------+------------+ -.. c:member:: long PyTypeObject.tp_weaklistoffset +.. c:member:: Py_ssize_t PyTypeObject.tp_weaklistoffset If the instances of this type are weakly referenceable, this field is greater than zero and contains the offset in the instance structure of the weak @@ -786,7 +786,7 @@ type objects) *must* have the :attr:`ob_size` field. .. XXX explain. -.. c:member:: long PyTypeObject.tp_dictoffset +.. c:member:: Py_ssize_t PyTypeObject.tp_dictoffset If the instances of this type have a dictionary containing instance variables, this field is non-zero and contains the offset in the instances of the type of diff --git a/Doc/extending/newtypes.rst b/Doc/extending/newtypes.rst index b7e35f4..f60e208 100644 --- a/Doc/extending/newtypes.rst +++ b/Doc/extending/newtypes.rst @@ -893,20 +893,20 @@ fields in the right order! It's often easiest to find an example that includes all the fields you need (even if they're initialized to ``0``) and then change the values to suit your new type. :: - char *tp_name; /* For printing */ + const char *tp_name; /* For printing */ The name of the type - as mentioned in the last section, this will appear in various places, almost entirely for diagnostic purposes. Try to choose something that will be helpful in such a situation! :: - int tp_basicsize, tp_itemsize; /* For allocation */ + Py_ssize_t tp_basicsize, tp_itemsize; /* For allocation */ These fields tell the runtime how much memory to allocate when new objects of this type are created. Python has some built-in support for variable length structures (think: strings, lists) which is where the :c:member:`~PyTypeObject.tp_itemsize` field comes in. This will be dealt with later. :: - char *tp_doc; + const char *tp_doc; Here you can put a string (or its address) that you want returned when the Python script references ``obj.__doc__`` to retrieve the doc string. diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst index 2d3cb1c..67a9c56 100644 --- a/Doc/faq/programming.rst +++ b/Doc/faq/programming.rst @@ -1164,6 +1164,8 @@ analogue of lisp car is ``lisp_list[0]`` and the analogue of cdr is usually a lot slower than using Python lists. +.. _faq-multidimensional-list: + How do I create a multidimensional list? ---------------------------------------- diff --git a/Doc/includes/typestruct.h b/Doc/includes/typestruct.h index 5c9f8f5..9f47899 100644 --- a/Doc/includes/typestruct.h +++ b/Doc/includes/typestruct.h @@ -1,7 +1,7 @@ typedef struct _typeobject { PyObject_VAR_HEAD - char *tp_name; /* For printing, in format "<module>.<name>" */ - int tp_basicsize, tp_itemsize; /* For allocation */ + const char *tp_name; /* For printing, in format "<module>.<name>" */ + Py_ssize_t tp_basicsize, tp_itemsize; /* For allocation */ /* Methods to implement standard operations */ @@ -9,7 +9,8 @@ typedef struct _typeobject { printfunc tp_print; getattrfunc tp_getattr; setattrfunc tp_setattr; - PyAsyncMethods *tp_as_async; + PyAsyncMethods *tp_as_async; /* formerly known as tp_compare (Python 2) + or tp_reserved (Python 3) */ reprfunc tp_repr; /* Method suites for standard classes */ @@ -30,9 +31,9 @@ typedef struct _typeobject { PyBufferProcs *tp_as_buffer; /* Flags to define presence of optional/expanded features */ - long tp_flags; + unsigned long tp_flags; - char *tp_doc; /* Documentation string */ + const char *tp_doc; /* Documentation string */ /* call function for all accessible objects */ traverseproc tp_traverse; @@ -44,7 +45,7 @@ typedef struct _typeobject { richcmpfunc tp_richcompare; /* weak reference enabler */ - long tp_weaklistoffset; + Py_ssize_t tp_weaklistoffset; /* Iterators */ getiterfunc tp_iter; @@ -58,7 +59,7 @@ typedef struct _typeobject { PyObject *tp_dict; descrgetfunc tp_descr_get; descrsetfunc tp_descr_set; - long tp_dictoffset; + Py_ssize_t tp_dictoffset; initproc tp_init; allocfunc tp_alloc; newfunc tp_new; @@ -69,7 +70,6 @@ typedef struct _typeobject { PyObject *tp_cache; PyObject *tp_subclasses; PyObject *tp_weaklist; - destructor tp_del; /* Type attribute cache version tag. Added in version 2.6 */ diff --git a/Doc/library/_thread.rst b/Doc/library/_thread.rst index a787e2f..7122861 100644 --- a/Doc/library/_thread.rst +++ b/Doc/library/_thread.rst @@ -93,7 +93,8 @@ It defines the following constants and functions: Return the thread stack size used when creating new threads. The optional *size* argument specifies the stack size to be used for subsequently created threads, and must be 0 (use platform or configured default) or a positive - integer value of at least 32,768 (32 KiB). If changing the thread stack size is + integer value of at least 32,768 (32 KiB). If *size* is not specified, + 0 is used. If changing the thread stack size is unsupported, a :exc:`RuntimeError` is raised. If the specified stack size is invalid, a :exc:`ValueError` is raised and the stack size is unmodified. 32 KiB is currently the minimum supported stack size value to guarantee sufficient diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst index 7d7caa8..146c8c9 100644 --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -6,7 +6,7 @@ Base Event Loop =============== The event loop is the central execution device provided by :mod:`asyncio`. -It provides multiple facilities, amongst which: +It provides multiple facilities, including: * Registering, executing and cancelling delayed calls (timeouts). diff --git a/Doc/library/asyncio-subprocess.rst b/Doc/library/asyncio-subprocess.rst index 3c9e3cb..c0704cd 100644 --- a/Doc/library/asyncio-subprocess.rst +++ b/Doc/library/asyncio-subprocess.rst @@ -303,7 +303,7 @@ Process .. _asyncio-subprocess-threads: Subprocess and threads -====================== +---------------------- asyncio supports running subprocesses from different threads, but there are limits: @@ -322,10 +322,10 @@ The :class:`asyncio.subprocess.Process` class is not thread safe. Subprocess examples -=================== +------------------- Subprocess using transport and protocol ---------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Example of a subprocess protocol using to get the output of a subprocess and to wait for the subprocess exit. The subprocess is created by the @@ -381,7 +381,7 @@ wait for the subprocess exit. The subprocess is created by the Subprocess using streams ------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^ Example using the :class:`~asyncio.subprocess.Process` class to control the subprocess and the :class:`StreamReader` class to read from the standard diff --git a/Doc/library/asyncio.rst b/Doc/library/asyncio.rst index b292ae9..9b4d65e 100644 --- a/Doc/library/asyncio.rst +++ b/Doc/library/asyncio.rst @@ -4,6 +4,13 @@ .. module:: asyncio :synopsis: Asynchronous I/O, event loop, coroutines and tasks. +.. note:: + + The asyncio package has been included in the standard library on a + :term:`provisional basis <provisional package>`. Backwards incompatible + changes (up to and including removal of the module) may occur if deemed + necessary by the core developers. + .. versionadded:: 3.4 **Source code:** :source:`Lib/asyncio/` diff --git a/Doc/library/binascii.rst b/Doc/library/binascii.rst index 3f7df74..e3f134b 100644 --- a/Doc/library/binascii.rst +++ b/Doc/library/binascii.rst @@ -59,7 +59,7 @@ The :mod:`binascii` module defines the following functions: should be at most 57 to adhere to the base64 standard. -.. function:: a2b_qp(string, header=False) +.. function:: a2b_qp(data, header=False) Convert a block of quoted-printable data back to binary and return the binary data. More than one line may be passed at a time. If the optional argument diff --git a/Doc/library/collections.rst b/Doc/library/collections.rst index f2befe7..8121cc4 100644 --- a/Doc/library/collections.rst +++ b/Doc/library/collections.rst @@ -842,10 +842,10 @@ field names, the method and attribute names start with an underscore. .. method:: somenamedtuple._asdict() Return a new :class:`OrderedDict` which maps field names to their corresponding - values. Note, this method is no longer needed now that the same effect can - be achieved by using the built-in :func:`vars` function:: + values:: - >>> vars(p) + >>> p = Point(x=11, y=22) + >>> p._asdict() OrderedDict([('x', 11), ('y', 22)]) .. versionchanged:: 3.1 @@ -1157,3 +1157,7 @@ attribute. be an instance of :class:`bytes`, :class:`str`, :class:`UserString` (or a subclass) or an arbitrary sequence which can be converted into a string using the built-in :func:`str` function. + + .. versionchanged:: 3.5 + New methods ``__getnewargs__``, ``__rmod__``, ``casefold``, + ``format_map``, ``isprintable``, and ``maketrans``. diff --git a/Doc/library/compileall.rst b/Doc/library/compileall.rst index 0325f1a..82f3753 100644 --- a/Doc/library/compileall.rst +++ b/Doc/library/compileall.rst @@ -88,14 +88,9 @@ compile Python sources. Added the ``-i``, ``-b`` and ``-h`` options. .. versionchanged:: 3.5 - Added the ``-j`` and ``-r`` options. - -.. versionchanged:: 3.5 - ``-q`` option was changed to a multilevel value. - -.. versionchanged:: 3.5 - ``-b`` will always produce a byte-code file ending in ``.pyc``, never - ``.pyo``. + Added the ``-j``, ``-r``, and ``-qq`` options. ``-q`` option + was changed to a multilevel value. ``b`` will always produce a + byte-code file ending in ``.pyc``, never ``.pyo``. There is no command-line option to control the optimization level used by the diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst index 7a214ed..1bcb3a4 100644 --- a/Doc/library/dis.rst +++ b/Doc/library/dis.rst @@ -946,8 +946,8 @@ the more significant byte last. Creates a new function object, sets its *__closure__* slot, and pushes it on the stack. TOS is the :term:`qualified name` of the function, TOS1 is the code associated with the function, and TOS2 is the tuple containing cells for - the closure's free variables. The function also has *argc* default - parameters, which are found below the cells. + the closure's free variables. *argc* is interpreted as in ``MAKE_FUNCTION``; + the annotations and defaults are also in the same order below TOS2. .. opcode:: BUILD_SLICE (argc) diff --git a/Doc/library/html.parser.rst b/Doc/library/html.parser.rst index b84c60b..824995e 100644 --- a/Doc/library/html.parser.rst +++ b/Doc/library/html.parser.rst @@ -185,7 +185,7 @@ implementations do nothing (except for :meth:`~HTMLParser.handle_startendtag`): The content of Internet Explorer conditional comments (condcoms) will also be sent to this method, so, for ``<!--[if IE 9]>IE9-specific content<![endif]-->``, - this method will receive ``'[if IE 9]>IE-specific content<![endif]'``. + this method will receive ``'[if IE 9]>IE9-specific content<![endif]'``. .. method:: HTMLParser.handle_decl(decl) diff --git a/Doc/library/imghdr.rst b/Doc/library/imghdr.rst index c60df24..f11f6dc 100644 --- a/Doc/library/imghdr.rst +++ b/Doc/library/imghdr.rst @@ -54,10 +54,8 @@ from :func:`what`: +------------+-----------------------------------+ .. versionadded:: 3.5 - The *exr* format was added. + The *exr* and *webp* formats were added. -.. versionchanged:: 3.5 - The *webp* type was added. You can extend the list of file types :mod:`imghdr` can recognize by appending to this variable: diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst index 3b62e7f..26a2913 100644 --- a/Doc/library/inspect.rst +++ b/Doc/library/inspect.rst @@ -178,6 +178,10 @@ attributes: +-----------+-----------------+---------------------------+ | | gi_code | code | +-----------+-----------------+---------------------------+ +| | gi_yieldfrom | object being iterated by | +| | | ``yield from``, or | +| | | ``None`` | ++-----------+-----------------+---------------------------+ | coroutine | __name__ | name | +-----------+-----------------+---------------------------+ | | __qualname__ | qualified name | @@ -191,10 +195,6 @@ attributes: +-----------+-----------------+---------------------------+ | | cr_code | code | +-----------+-----------------+---------------------------+ -| | gi_yieldfrom | object being iterated by | -| | | ``yield from``, or | -| | | ``None`` | -+-----------+-----------------+---------------------------+ | builtin | __doc__ | documentation string | +-----------+-----------------+---------------------------+ | | __name__ | original name of this | @@ -209,9 +209,10 @@ attributes: .. versionchanged:: 3.5 - Add ``__qualname__`` attribute to generators. The ``__name__`` attribute of - generators is now set from the function name, instead of the code name, and - it can now be modified. + Add ``__qualname__`` and ``gi_yieldfrom`` attributes to generators. + + The ``__name__`` attribute of generators is now set from the function + name, instead of the code name, and it can now be modified. .. function:: getmembers(object[, predicate]) @@ -1038,6 +1039,11 @@ line. returned list represents *frame*; the last entry represents the outermost call on *frame*'s stack. + .. versionchanged:: 3.5 + A list of :term:`named tuples <named tuple>` + ``FrameInfo(frame, filename, lineno, function, code_context, index)`` + is returned. + .. function:: getinnerframes(traceback, context=1) @@ -1046,6 +1052,11 @@ line. list represents *traceback*; the last entry represents where the exception was raised. + .. versionchanged:: 3.5 + A list of :term:`named tuples <named tuple>` + ``FrameInfo(frame, filename, lineno, function, code_context, index)`` + is returned. + .. function:: currentframe() @@ -1065,6 +1076,11 @@ line. returned list represents the caller; the last entry represents the outermost call on the stack. + .. versionchanged:: 3.5 + A list of :term:`named tuples <named tuple>` + ``FrameInfo(frame, filename, lineno, function, code_context, index)`` + is returned. + .. function:: trace(context=1) @@ -1073,6 +1089,11 @@ line. entry in the list represents the caller; the last entry represents where the exception was raised. + .. versionchanged:: 3.5 + A list of :term:`named tuples <named tuple>` + ``FrameInfo(frame, filename, lineno, function, code_context, index)`` + is returned. + Fetching attributes statically ------------------------------ diff --git a/Doc/library/ipaddress.rst b/Doc/library/ipaddress.rst index 7b59440..90fcc74 100644 --- a/Doc/library/ipaddress.rst +++ b/Doc/library/ipaddress.rst @@ -575,7 +575,7 @@ so to avoid duplication they are only documented for :class:`IPv4Network`. single-address network, with the network address being *address* and the mask being ``/128``. - 3. An integer packed into a :class:`bytes` object of length 16, bit-endian. + 3. An integer packed into a :class:`bytes` object of length 16, big-endian. The interpretation is similar to an integer *address*. 4. A two-tuple of an address description and a netmask, where the address diff --git a/Doc/library/os.path.rst b/Doc/library/os.path.rst index e4fe44e..a3fe73c 100644 --- a/Doc/library/os.path.rst +++ b/Doc/library/os.path.rst @@ -82,8 +82,21 @@ the :mod:`glob` module.) Return the longest path prefix (taken character-by-character) that is a prefix of all paths in *list*. If *list* is empty, return the empty string - (``''``). Note that this may return invalid paths because it works a - character at a time. To obtain a valid path, see :func:`commonpath`. + (``''``). + + .. note:: + + This function may return invalid paths because it works a + character at a time. To obtain a valid path, see + :func:`commonpath`. + + :: + + >>> os.path.commonprefix(['/usr/lib', '/usr/local/lib']) + '/usr/l' + + >>> os.path.commonpath(['/usr/lib', '/usr/local/lib']) + '/usr' .. function:: dirname(path) diff --git a/Doc/library/os.rst b/Doc/library/os.rst index d4032e0..13f83c6 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -857,9 +857,9 @@ as internal buffering of data. :data:`os.SEEK_HOLE` or :data:`os.SEEK_DATA`. -.. function:: open(file, flags, mode=0o777, *, dir_fd=None) +.. function:: open(path, flags, mode=0o777, *, dir_fd=None) - Open the file *file* and set various flags according to *flags* and possibly + Open the file *path* and set various flags according to *flags* and possibly its mode according to *mode*. When computing *mode*, the current umask value is first masked out. Return the file descriptor for the newly opened file. The new file descriptor is :ref:`non-inheritable <fd_inheritance>`. @@ -1071,10 +1071,10 @@ or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Window :exc:`InterruptedError` exception (see :pep:`475` for the rationale). -.. function:: sendfile(out, in, offset, nbytes) - sendfile(out, in, offset, nbytes, headers=None, trailers=None, flags=0) +.. function:: sendfile(out, in, offset, count) + sendfile(out, in, offset, count, [headers], [trailers], flags=0) - Copy *nbytes* bytes from file descriptor *in* to file descriptor *out* + Copy *count* bytes from file descriptor *in* to file descriptor *out* starting at *offset*. Return the number of bytes sent. When EOF is reached return 0. @@ -1088,7 +1088,7 @@ or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Window *trailers* are arbitrary sequences of buffers that are written before and after the data from *in* is written. It returns the same as the first case. - On Mac OS X and FreeBSD, a value of 0 for *nbytes* specifies to send until + On Mac OS X and FreeBSD, a value of 0 for *count* specifies to send until the end of *in* is reached. All platforms support sockets as *out* file descriptor, and some platforms @@ -1690,10 +1690,10 @@ features: The *dir_fd* argument. -.. function:: mknod(filename, mode=0o600, device=0, *, dir_fd=None) +.. function:: mknod(path, mode=0o600, device=0, *, dir_fd=None) Create a filesystem node (file, device special file or named pipe) named - *filename*. *mode* specifies both the permissions to use and the type of node + *path*. *mode* specifies both the permissions to use and the type of node to be created, being combined (bitwise OR) with one of ``stat.S_IFREG``, ``stat.S_IFCHR``, ``stat.S_IFBLK``, and ``stat.S_IFIFO`` (those constants are available in :mod:`stat`). For ``stat.S_IFCHR`` and ``stat.S_IFBLK``, @@ -2389,9 +2389,9 @@ features: .. versionadded:: 3.3 -.. function:: symlink(source, link_name, target_is_directory=False, *, dir_fd=None) +.. function:: symlink(src, dst, target_is_directory=False, *, dir_fd=None) - Create a symbolic link pointing to *source* named *link_name*. + Create a symbolic link pointing to *src* named *dst*. On Windows, a symlink represents either a file or a directory, and does not morph to the target dynamically. If the target is present, the type of the @@ -2461,20 +2461,20 @@ features: The *dir_fd* parameter. -.. function:: utime(path, times=None, *, ns=None, dir_fd=None, follow_symlinks=True) +.. function:: utime(path, times=None, *[, ns], dir_fd=None, follow_symlinks=True) Set the access and modified times of the file specified by *path*. :func:`utime` takes two optional parameters, *times* and *ns*. These specify the times set on *path* and are used as follows: - - If *ns* is not ``None``, + - If *ns* is specified, it must be a 2-tuple of the form ``(atime_ns, mtime_ns)`` where each member is an int expressing nanoseconds. - If *times* is not ``None``, it must be a 2-tuple of the form ``(atime, mtime)`` where each member is an int or float expressing seconds. - - If *times* and *ns* are both ``None``, + - If *times* is ``None`` and *ns* is unspecified, this is equivalent to specifying ``ns=(atime_ns, mtime_ns)`` where both times are the current time. @@ -3023,9 +3023,10 @@ written in Python, such as a mail server's external command delivery program. Availability: Unix. -.. function:: popen(command, mode='r', buffering=-1) +.. function:: popen(cmd, mode='r', buffering=-1) - Open a pipe to or from *command*. The return value is an open file object + Open a pipe to or from command *cmd*. + The return value is an open file object connected to the pipe, which can be read or written depending on whether *mode* is ``'r'`` (default) or ``'w'``. The *buffering* argument has the same meaning as the corresponding argument to the built-in :func:`open` function. The diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst index 1dcdb2f..3ecab9e 100644 --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -106,8 +106,30 @@ created. Socket addresses are represented as follows: .. versionadded:: 3.3 -- Certain other address families (:const:`AF_BLUETOOTH`, :const:`AF_PACKET`, - :const:`AF_CAN`) support specific representations. +- :const:`AF_BLUETOOTH` supports the following protocols and address + formats: + + - :const:`BTPROTO_L2CAP` accepts ``(bdaddr, psm)`` where ``bdaddr`` is + the Bluetooth address as a string and ``psm`` is an integer. + + - :const:`BTPROTO_RFCOMM` accepts ``(bdaddr, channel)`` where ``bdaddr`` + is the Bluetooth address as a string and ``channel`` is an integer. + + - :const:`BTPROTO_HCI` accepts ``(device_id,)`` where ``device_id`` is + either an integer or a string with the Bluetooth address of the + interface. (This depends on your OS; NetBSD and DragonFlyBSD expect + a Bluetooth address while everything else expects an integer.) + + .. versionchanged:: 3.2 + NetBSD and DragonFlyBSD support added. + + - :const:`BTPROTO_SCO` accepts ``bdaddr`` where ``bdaddr`` is a + :term:`bytes-like object` containing the Bluetooth address in a + string format. (ex. ``b'12:23:34:45:56:67'``) This protocol is not + supported under FreeBSD. + +- Certain other address families (:const:`AF_PACKET`, :const:`AF_CAN`) + support specific representations. .. XXX document them! diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index 8573416..b3c9e23 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -854,8 +854,8 @@ operations have the same priority as the corresponding numeric operations. | ``s + t`` | the concatenation of *s* and | (6)(7) | | | *t* | | +--------------------------+--------------------------------+----------+ -| ``s * n`` or | *n* shallow copies of *s* | (2)(7) | -| ``n * s`` | concatenated | | +| ``s * n`` or | equivalent to adding *s* to | (2)(7) | +| ``n * s`` | itself *n* times | | +--------------------------+--------------------------------+----------+ | ``s[i]`` | *i*\ th item of *s*, origin 0 | \(3) | +--------------------------+--------------------------------+----------+ @@ -897,9 +897,9 @@ Notes: (2) Values of *n* less than ``0`` are treated as ``0`` (which yields an empty - sequence of the same type as *s*). Note also that the copies are shallow; - nested structures are not copied. This often haunts new Python programmers; - consider:: + sequence of the same type as *s*). Note that items in the sequence *s* + are not copied; they are referenced multiple times. This often haunts + new Python programmers; consider:: >>> lists = [[]] * 3 >>> lists @@ -909,7 +909,7 @@ Notes: [[3], [3], [3]] What has happened is that ``[[]]`` is a one-element list containing an empty - list, so all three elements of ``[[]] * 3`` are (pointers to) this single empty + list, so all three elements of ``[[]] * 3`` are references to this single empty list. Modifying any of the elements of ``lists`` modifies this single list. You can create a list of different lists this way:: @@ -920,6 +920,9 @@ Notes: >>> lists [[3], [5], [7]] + Further explanation is available in the FAQ entry + :ref:`faq-multidimensional-list`. + (3) If *i* or *j* is negative, the index is relative to the end of the string: ``len(s) + i`` or ``len(s) + j`` is substituted. But note that ``-0`` is diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst index efcacce..adf99ec 100644 --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -1068,7 +1068,7 @@ Return code handling translates as follows:: if rc is not None and rc >> 8: print("There were some errors") ==> - process = Popen(cmd, 'w', stdin=PIPE) + process = Popen(cmd, stdin=PIPE) ... process.stdin.close() if process.wait() != 0: diff --git a/Doc/library/tempfile.rst b/Doc/library/tempfile.rst index 0387fb1..83f9941 100644 --- a/Doc/library/tempfile.rst +++ b/Doc/library/tempfile.rst @@ -16,16 +16,18 @@ -------------- -This module generates temporary files and directories. It works on all -supported platforms. It provides three new functions, -:func:`NamedTemporaryFile`, :func:`mkstemp`, and :func:`mkdtemp`, which should -eliminate all remaining need to use the insecure :func:`mktemp` function. -Temporary file names created by this module no longer contain the process ID; -instead a string of six random characters is used. - -Also, all the user-callable functions now take additional arguments which -allow direct control over the location and name of temporary files. It is -no longer necessary to use the global *tempdir* variable. +This module creates temporary files and directories. It works on all +supported platforms. :class:`TemporaryFile`, :class:`NamedTemporaryFile`, +:class:`TemporaryDirectory`, and :class:`SpooledTemporaryFile` are high-level +interfaces which provide automatic cleanup and can be used as +context managers. :func:`mkstemp` and +:func:`mkdtemp` are lower-level functions which require manual cleanup. + +All the user-callable functions and constructors take additional arguments which +allow direct control over the location and name of temporary files and +directories. Files names used by this module include a string of +random characters which allows those files to be securely created in +shared temporary directories. To maintain backward compatibility, the argument order is somewhat odd; it is recommended to use keyword arguments for clarity. @@ -34,28 +36,33 @@ The module defines the following user-callable items: .. function:: TemporaryFile(mode='w+b', buffering=None, encoding=None, newline=None, suffix='', prefix='tmp', dir=None) Return a :term:`file-like object` that can be used as a temporary storage area. - The file is created using :func:`mkstemp`. It will be destroyed as soon + The file is created securely, using the same rules as :func:`mkstemp`. It will be destroyed as soon as it is closed (including an implicit close when the object is garbage - collected). Under Unix, the directory entry for the file is removed + collected). Under Unix, the directory entry for the file is either not created at all or is removed immediately after the file is created. Other platforms do not support this; your code should not rely on a temporary file created using this function having or not having a visible name in the file system. + The resulting object can be used as a context manager (see + :ref:`tempfile-examples`). On completion of the context or + destruction of the file object the temporary file will be removed + from the filesystem. + The *mode* parameter defaults to ``'w+b'`` so that the file created can be read and written without being closed. Binary mode is used so that it behaves consistently on all platforms without regard for the data that is stored. *buffering*, *encoding* and *newline* are interpreted as for :func:`open`. - The *dir*, *prefix* and *suffix* parameters are passed to :func:`mkstemp`. + The *dir*, *prefix* and *suffix* parameters have the same meaning + as with :func:`mkstemp`. The returned object is a true file object on POSIX platforms. On other platforms, it is a file-like object whose :attr:`!file` attribute is the - underlying true file object. This file-like object can be used in a - :keyword:`with` statement, just like a normal file. + underlying true file object. The :py:data:`os.O_TMPFILE` flag is used if it is available and works - (Linux-specific, require Linux kernel 3.11 or later). + (Linux-specific, requires Linux kernel 3.11 or later). .. versionchanged:: 3.5 @@ -101,10 +108,9 @@ The module defines the following user-callable items: .. function:: TemporaryDirectory(suffix='', prefix='tmp', dir=None) - This function creates a temporary directory using :func:`mkdtemp` - (the supplied arguments are passed directly to the underlying function). + This function securely creates a temporary directory using the same rules as :func:`mkdtemp`. The resulting object can be used as a context manager (see - :ref:`context-managers`). On completion of the context or destruction + :ref:`tempfile-examples`). On completion of the context or destruction of the temporary directory object the newly created temporary directory and all its contents are removed from the filesystem. @@ -194,49 +200,14 @@ The module defines the following user-callable items: an appropriate default value to be used. -.. function:: mktemp(suffix='', prefix='tmp', dir=None) - - .. deprecated:: 2.3 - Use :func:`mkstemp` instead. - - Return an absolute pathname of a file that did not exist at the time the - call is made. The *prefix*, *suffix*, and *dir* arguments are the same - as for :func:`mkstemp`. - - .. warning:: - - Use of this function may introduce a security hole in your program. By - the time you get around to doing anything with the file name it returns, - someone else may have beaten you to the punch. :func:`mktemp` usage can - be replaced easily with :func:`NamedTemporaryFile`, passing it the - ``delete=False`` parameter:: - - >>> f = NamedTemporaryFile(delete=False) - >>> f.name - '/tmp/tmptjujjt' - >>> f.write(b"Hello World!\n") - 13 - >>> f.close() - >>> os.unlink(f.name) - >>> os.path.exists(f.name) - False - -The module uses a global variable that tell it how to construct a -temporary name. They are initialized at the first call to any of the -functions above. The caller may change them, but this is discouraged; use -the appropriate function arguments, instead. - +.. function:: gettempdir() -.. data:: tempdir + Return the name of the directory used for temporary files. This + defines the default value for the *dir* argument to all functions + in this module. - When set to a value other than ``None``, this variable defines the - default value for the *dir* argument to all the functions defined in this - module. - - If ``tempdir`` is unset or ``None`` at any call to any of the above - functions, Python searches a standard list of directories and sets - *tempdir* to the first one which the calling user can create files in. - The list is: + Python searches a standard list of directories to find one which + the calling user can create files in. The list is: #. The directory named by the :envvar:`TMPDIR` environment variable. @@ -254,12 +225,8 @@ the appropriate function arguments, instead. #. As a last resort, the current working directory. - -.. function:: gettempdir() - - Return the directory currently selected to create temporary files in. If - :data:`tempdir` is not ``None``, this simply returns its contents; otherwise, - the search described above is performed, and the result returned. + The result of this search is cached, see the description of + :data:`tempdir` below. .. function:: gettempdirb() @@ -278,6 +245,23 @@ the appropriate function arguments, instead. .. versionadded:: 3.5 +The module uses a global variable to store the name of the directory +used for temporary files returned by :func:`gettempdir`. It can be +set directly to override the selection process, but this is discouraged. +All functions in this module take a *dir* argument which can be used +to specify the directory and this is the recommend approach. + +.. data:: tempdir + + When set to a value other than ``None``, this variable defines the + default value for the *dir* argument to all the functions defined in this + module. + + If ``tempdir`` is unset or ``None`` at any call to any of the above + functions except :func:`gettempprefix` it is initalized following the + algorithm described in :func:`gettempdir`. + +.. _tempfile-examples: Examples -------- @@ -311,3 +295,42 @@ Here are some examples of typical usage of the :mod:`tempfile` module:: >>> # directory and contents have been removed + +Deprecated functions and variables +---------------------------------- + +A historical way to create temporary files was to first generate a +file name with the :func:`mktemp` function and then create a file +using this name. Unfortunately this is not secure, because a different +process may create a file with this name in the time between the call +to :func:`mktemp` and the subsequent attempt to create the file by the +first process. The solution is to combine the two steps and create the +file immediately. This approach is used by :func:`mkstemp` and the +other functions described above. + +.. function:: mktemp(suffix='', prefix='tmp', dir=None) + + .. deprecated:: 2.3 + Use :func:`mkstemp` instead. + + Return an absolute pathname of a file that did not exist at the time the + call is made. The *prefix*, *suffix*, and *dir* arguments are the same + as for :func:`mkstemp`. + + .. warning:: + + Use of this function may introduce a security hole in your program. By + the time you get around to doing anything with the file name it returns, + someone else may have beaten you to the punch. :func:`mktemp` usage can + be replaced easily with :func:`NamedTemporaryFile`, passing it the + ``delete=False`` parameter:: + + >>> f = NamedTemporaryFile(delete=False) + >>> f.name + '/tmp/tmptjujjt' + >>> f.write(b"Hello World!\n") + 13 + >>> f.close() + >>> os.unlink(f.name) + >>> os.path.exists(f.name) + False diff --git a/Doc/library/threading.rst b/Doc/library/threading.rst index 5269994..c56d707 100644 --- a/Doc/library/threading.rst +++ b/Doc/library/threading.rst @@ -89,7 +89,8 @@ This module defines the following functions: Return the thread stack size used when creating new threads. The optional *size* argument specifies the stack size to be used for subsequently created threads, and must be 0 (use platform or configured default) or a positive - integer value of at least 32,768 (32 KiB). If changing the thread stack size is + integer value of at least 32,768 (32 KiB). If *size* is not specified, + 0 is used. If changing the thread stack size is unsupported, a :exc:`RuntimeError` is raised. If the specified stack size is invalid, a :exc:`ValueError` is raised and the stack size is unmodified. 32 KiB is currently the minimum supported stack size value to guarantee sufficient diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index 8089443..e09b647 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -68,7 +68,7 @@ subscription to denote expected types for container elements. overrides: Mapping[str, str]) -> None: ... Generics can be parametrized by using a new factory available in typing -called :class:`TypeVar`. +called TypeVar. .. code-block:: python @@ -145,7 +145,7 @@ This is thus invalid:: class Pair(Generic[T, T]): # INVALID ... -You can use multiple inheritance with :class:`Generic`:: +You can use multiple inheritance with `Generic`:: from typing import TypeVar, Generic, Sized @@ -154,17 +154,6 @@ You can use multiple inheritance with :class:`Generic`:: class LinkedList(Sized, Generic[T]): ... -When inheriting from generic classes, some type variables could fixed:: - - from typing import TypeVar, Mapping - - T = TypeVar('T') - - class MyDict(Mapping[str, T]): - ... - -In this case ``MyDict`` has a single parameter, ``T``. - Subclassing a generic class without specifying type parameters assumes :class:`Any` for each position. In the following example, ``MyIterable`` is not generic but implicitly inherits from ``Iterable[Any]``:: @@ -173,11 +162,7 @@ not generic but implicitly inherits from ``Iterable[Any]``:: class MyIterable(Iterable): # Same as Iterable[Any] -The metaclass used by :class:`Generic` is a subclass of :class:`abc.ABCMeta`. -A generic class can be an ABC by including abstract methods or properties, -and generic classes can also have ABCs as base classes without a metaclass -conflict. Generic metaclasses are not supported. - +Generic metaclasses are not supported. The :class:`Any` type --------------------- @@ -193,6 +178,15 @@ when a value has type :class:`Any`, the type checker will allow all operations on it, and a value of type :class:`Any` can be assigned to a variable (or used as a return value) of a more constrained type. +Default argument values +----------------------- + +Use a literal ellipsis ``...`` to declare an argument as having a default value:: + + from typing import AnyStr + + def foo(x: AnyStr, y: AnyStr = ...) -> AnyStr: ... + Classes, functions, and decorators ---------------------------------- @@ -242,11 +236,7 @@ The module defines the following classes, functions and decorators: Type variables may be marked covariant or contravariant by passing ``covariant=True`` or ``contravariant=True``. See :pep:`484` for more - details. By default type variables are invariant. Alternatively, - a type variable may specify an upper bound using ``bound=<type>``. - This means that an actual type substituted (explicitly or implictly) - for the type variable must be a subclass of the boundary type, - see :pep:`484`. + details. By default type variables are invariant. .. class:: Union @@ -339,139 +329,57 @@ The module defines the following classes, functions and decorators: .. class:: Iterable(Generic[T_co]) - A generic version of the :class:`collections.abc.Iterable`. - .. class:: Iterator(Iterable[T_co]) - A generic version of the :class:`collections.abc.Iterator`. - .. class:: SupportsInt - An ABC with one abstract method `__int__`. - .. class:: SupportsFloat - An ABC with one abstract method `__float__`. - .. class:: SupportsAbs - An ABC with one abstract method `__abs__` that is covariant - in its return type. - .. class:: SupportsRound - An ABC with one abstract method `__round__` - that is covariant in its return type. - .. class:: Reversible - An ABC with one abstract method `__reversed__` returning - an `Iterator[T_co]`. - .. class:: Container(Generic[T_co]) - A generic version of :class:`collections.abc.Container`. - .. class:: AbstractSet(Sized, Iterable[T_co], Container[T_co]) - A generic version of :class:`collections.abc.Set`. - .. class:: MutableSet(AbstractSet[T]) - A generic version of :class:`collections.abc.MutableSet`. - -.. class:: Mapping(Sized, Iterable[KT], Container[KT], Generic[VT_co]) - - A generic version of :class:`collections.abc.Mapping`. +.. class:: Mapping(Sized, Iterable[KT_co], Container[KT_co], Generic[KT_co, VT_co]) .. class:: MutableMapping(Mapping[KT, VT]) - A generic version of :class:`collections.abc.MutableMapping`. - .. class:: Sequence(Sized, Iterable[T_co], Container[T_co]) - A generic version of :class:`collections.abc.Sequence`. - .. class:: MutableSequence(Sequence[T]) - A generic version of :class:`collections.abc.MutableSequence`. - .. class:: ByteString(Sequence[int]) - A generic version of :class:`collections.abc.ByteString`. - - This type represents the types :class:`bytes`, :class:`bytearray`, - and :class:`memoryview`. - - As a shorthand for this type, :class:`bytes` can be used to - annotate arguments of any of the types mentioned above. - .. class:: List(list, MutableSequence[T]) - Generic version of :class:`list`. - Useful for annotating return types. To annotate arguments it is preferred - to use abstract collection types such as :class:`Mapping`, :class:`Sequence`, - or :class:`AbstractSet`. - - This type may be used as follows:: - - T = TypeVar('T', int, float) - - def vec2(x: T, y: T) -> List[T]: - return [x, y] - - def slice__to_4(vector: Sequence[T]) -> List[T]: - return vector[0:4] - -.. class:: AbstractSet(set, MutableSet[T]) - - A generic version of :class:`collections.abc.Set`. +.. class:: Set(set, MutableSet[T]) .. class:: MappingView(Sized, Iterable[T_co]) - A generic version of :class:`collections.abc.MappingView`. - .. class:: KeysView(MappingView[KT_co], AbstractSet[KT_co]) - A generic version of :class:`collections.abc.KeysView`. - .. class:: ItemsView(MappingView, Generic[KT_co, VT_co]) - A generic version of :class:`collections.abc.ItemsView`. - .. class:: ValuesView(MappingView[VT_co]) - A generic version of :class:`collections.abc.ValuesView`. - .. class:: Dict(dict, MutableMapping[KT, VT]) - A generic version of :class:`dict`. - The usage of this type is as follows:: - - def get_position_in_index(word_list: Dict[str, int], word: str) -> int: - return word_list[word] - .. class:: Generator(Iterator[T_co], Generic[T_co, T_contra, V_co]) .. class:: io - Wrapper namespace for I/O stream types. - - This defines the generic type ``IO[AnyStr]`` and aliases ``TextIO`` - and ``BinaryIO`` for respectively ``IO[str]`` and ``IO[bytes]``. - These representing the types of I/O streams such as returned by - :func:`open`. + Wrapper namespace for IO generic classes. .. class:: re - Wrapper namespace for regular expression matching types. - - This defines the type aliases ``Pattern`` and ``Match`` which - correspond to the return types from :func:`re.compile` and - :func:`re.match`. These types (and the corresponding functions) - are generic in ``AnyStr`` and can be made specific by writing - ``Pattern[str]``, ``Pattern[bytes]``, ``Match[str]``, or - ``Match[bytes]``. + Wrapper namespace for re type classes. .. function:: NamedTuple(typename, fields) diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst index 5085703..1153f83 100644 --- a/Doc/library/unittest.rst +++ b/Doc/library/unittest.rst @@ -278,8 +278,8 @@ The :option:`-s`, :option:`-p`, and :option:`-t` options can be passed in as positional arguments in that order. The following two command lines are equivalent:: - python -m unittest discover -s project_directory -p '*_test.py' - python -m unittest discover project_directory '*_test.py' + python -m unittest discover -s project_directory -p "*_test.py" + python -m unittest discover project_directory "*_test.py" As well as being a path it is possible to pass a package name, for example ``myproject.subpackage.test``, as the start directory. The package name you diff --git a/Doc/library/xml.etree.elementtree.rst b/Doc/library/xml.etree.elementtree.rst index eef1b58..14e5c99 100644 --- a/Doc/library/xml.etree.elementtree.rst +++ b/Doc/library/xml.etree.elementtree.rst @@ -651,21 +651,29 @@ Element Objects .. attribute:: text + tail - The *text* attribute can be used to hold additional data associated with - the element. As the name implies this attribute is usually a string but - may be any application-specific object. If the element is created from - an XML file the attribute will contain any text found between the element - tags. + These attributes can be used to hold additional data associated with + the element. Their values are usually strings but may be any + application-specific object. If the element is created from + an XML file, the *text* attribute holds either the text between + the element's start tag and its first child or end tag, or ``None``, and + the *tail* attribute holds either the text between the element's + end tag and the next tag, or ``None``. For the XML data + .. code-block:: xml - .. attribute:: tail + <a><b>1<c>2<d/>3</c></b>4</a> - The *tail* attribute can be used to hold additional data associated with - the element. This attribute is usually a string but may be any - application-specific object. If the element is created from an XML file - the attribute will contain any text found after the element's end tag and - before the next tag. + the *a* element has ``None`` for both *text* and *tail* attributes, + the *b* element has *text* ``"1"`` and *tail* ``"4"``, + the *c* element has *text* ``"2"`` and *tail* ``None``, + and the *d* element has *text* ``None`` and *tail* ``"3"``. + + To collect the inner text of an element, see :meth:`itertext`, for + example ``"".join(element.itertext())``. + + Applications may store arbitrary objects in these attributes. .. attribute:: attrib diff --git a/Doc/library/zlib.rst b/Doc/library/zlib.rst index b178fe1..aea0e79 100644 --- a/Doc/library/zlib.rst +++ b/Doc/library/zlib.rst @@ -58,7 +58,7 @@ The available exception and functions in this module are: Raises the :exc:`error` exception if any error occurs. -.. function:: compressobj(level=-1, method=DEFLATED, wbits=15, memlevel=8, strategy=Z_DEFAULT_STRATEGY[, zdict]) +.. function:: compressobj(level=-1, method=DEFLATED, wbits=15, memLevel=8, strategy=Z_DEFAULT_STRATEGY[, zdict]) Returns a compression object, to be used for compressing data streams that won't fit into memory at once. @@ -75,9 +75,9 @@ The available exception and functions in this module are: should be an integer from ``8`` to ``15``. Higher values give better compression, but use more memory. - *memlevel* controls the amount of memory used for internal compression state. - Valid values range from ``1`` to ``9``. Higher values using more memory, - but are faster and produce smaller output. + The *memLevel* argument controls the amount of memory used for the + internal compression state. Valid values range from ``1`` to ``9``. + Higher values use more memory, but are faster and produce smaller output. *strategy* is used to tune the compression algorithm. Possible values are ``Z_DEFAULT_STRATEGY``, ``Z_FILTERED``, and ``Z_HUFFMAN_ONLY``. diff --git a/Doc/tools/susp-ignored.csv b/Doc/tools/susp-ignored.csv index 7664f92..ece411e 100644 --- a/Doc/tools/susp-ignored.csv +++ b/Doc/tools/susp-ignored.csv @@ -288,6 +288,6 @@ library/xml.etree.elementtree,332,:character,"for char in actor.findall('role:ch library/zipapp,31,:main,"$ python -m zipapp myapp -m ""myapp:main""" library/zipapp,82,:fn,"argument should have the form ""pkg.mod:fn"", where ""pkg.mod"" is a" library/zipapp,155,:callable,"""pkg.module:callable"" and the archive will be run by importing" -library/stdtypes,3767,::,>>> m[::2].tolist() +library/stdtypes,,::,>>> m[::2].tolist() library/sys,1115,`,# ``wrapper`` creates a ``wrap(coro)`` coroutine: tutorial/venv,77,:c7b9645a6f35,"Python 3.4.3+ (3.4:c7b9645a6f35+, May 22 2015, 09:31:25)" diff --git a/Doc/tutorial/datastructures.rst b/Doc/tutorial/datastructures.rst index a2031ed..0d51480 100644 --- a/Doc/tutorial/datastructures.rst +++ b/Doc/tutorial/datastructures.rst @@ -612,18 +612,18 @@ returns a new sorted list while leaving the source unaltered. :: orange pear -To change a sequence you are iterating over while inside the loop (for -example to duplicate certain items), it is recommended that you first make -a copy. Looping over a sequence does not implicitly make a copy. The slice -notation makes this especially convenient:: - - >>> words = ['cat', 'window', 'defenestrate'] - >>> for w in words[:]: # Loop over a slice copy of the entire list. - ... if len(w) > 6: - ... words.insert(0, w) +It is sometimes tempting to change a list while you are looping over it; +however, it is often simpler and safer to create a new list instead. :: + + >>> import math + >>> raw_data = [56.2, float('NaN'), 51.7, 55.3, 52.5, float('NaN'), 47.8] + >>> filtered_data = [] + >>> for value in raw_data: + ... if not math.isnan(value): + ... filtered_data.append(value) ... - >>> words - ['defenestrate', 'cat', 'window', 'defenestrate'] + >>> filtered_data + [56.2, 51.7, 55.3, 52.5, 47.8] .. _tut-conditions: diff --git a/Doc/using/mac.rst b/Doc/using/mac.rst index ede864d..ef091e5 100644 --- a/Doc/using/mac.rst +++ b/Doc/using/mac.rst @@ -102,8 +102,9 @@ Configuration Python on OS X honors all standard Unix environment variables such as :envvar:`PYTHONPATH`, but setting these variables for programs started from the Finder is non-standard as the Finder does not read your :file:`.profile` or -:file:`.cshrc` at startup. You need to create a file :file:`~ -/.MacOSX/environment.plist`. See Apple's Technical Document QA1067 for details. +:file:`.cshrc` at startup. You need to create a file +:file:`~/.MacOSX/environment.plist`. See Apple's Technical Document QA1067 for +details. For more information on installation Python packages in MacPython, see section :ref:`mac-package-manager`. diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst index 11f871b..aec6da0 100644 --- a/Doc/whatsnew/3.5.rst +++ b/Doc/whatsnew/3.5.rst @@ -4,6 +4,7 @@ :Release: |release| :Date: |today| +:Editor: Elvis Pranskevichus <elprans@gmail.com> .. Rules for maintenance: @@ -46,7 +47,6 @@ when researching a change. This article explains the new features in Python 3.5, compared to 3.4. - For full details, see the :source:`Misc/NEWS` file. .. note:: @@ -81,13 +81,16 @@ New library modules: New built-in features: * ``bytes % args``, ``bytearray % args``: :pep:`461` - Adding ``%`` formatting - to bytes and bytearray + to bytes and bytearray. + * ``b'\xf0\x9f\x90\x8d'.hex()``, ``bytearray(b'\xf0\x9f\x90\x8d').hex()``, ``memoryview(b'\xf0\x9f\x90\x8d').hex()``: :issue:`9951` - A ``hex`` method has been added to bytes, bytearray, and memoryview. + * Generators have new ``gi_yieldfrom`` attribute, which returns the object being iterated by ``yield from`` expressions. (Contributed by Benno Leslie and Yury Selivanov in :issue:`24450`.) + * New :exc:`RecursionError` exception. (Contributed by Georg Brandl in :issue:`19235`.) @@ -99,6 +102,7 @@ Implementation improvements: (:issue:`19977`). * :pep:`488`, the elimination of ``.pyo`` files. + * :pep:`489`, multi-phase initialization of extension modules. Significantly Improved Library Modules: @@ -108,147 +112,256 @@ Significantly Improved Library Modules: :issue:`16991`. * You may now pass bytes to the :mod:`tempfile` module's APIs and it will - return the temporary pathname as bytes instead of str. It also accepts - a value of ``None`` on parameters where only str was accepted in the past to - do the right thing based on the types of the other inputs. Two functions, - :func:`gettempdirb` and :func:`gettempprefixb`, have been added to go along - with this. This behavior matches that of the :mod:`os` APIs. + return the temporary pathname as :class:`bytes` instead of :class:`str`. + It also accepts a value of ``None`` on parameters where only str was + accepted in the past to do the right thing based on the types of the + other inputs. Two functions, :func:`gettempdirb` and + :func:`gettempprefixb`, have been added to go along with this. + This behavior matches that of the :mod:`os` APIs. + (Contributed by Gregory P. Smith in :issue:`24230`.) * :mod:`ssl` module gained support for Memory BIO, which decouples SSL protocol handling from network IO. (Contributed by Geert Jansen in :issue:`21965`.) +* :mod:`traceback` has new lightweight and convenient to work with + classes :class:`~traceback.TracebackException`, + :class:`~traceback.StackSummary`, and :class:`traceback.FrameSummary`. + (Contributed by Robert Collins in :issue:`17911`.) + +* Most of :func:`functools.lru_cache` machinery is now implemented in C. + (Contributed by Matt Joiner, Alexey Kachayev, and Serhiy Storchaka + in :issue:`14373`.) + Security improvements: -* None yet. +* SSLv3 is now disabled throughout the standard library. + It can still be enabled by instantiating a :class:`ssl.SSLContext` + manually. (See :issue:`22638` for more details; this change was + backported to CPython 3.4 and 2.7.) + +* HTTP cookie parsing is now stricter, in order to protect + against potential injection attacks. (Contributed by Antoine Pitrou + in :issue:`22796`.) Windows improvements: * A new installer for Windows has replaced the old MSI. See :ref:`using-on-windows` for more information. + * Windows builds now use Microsoft Visual C++ 14.0, and extension modules should use the same. +Please read on for a comprehensive list of user-facing changes, including many +other smaller improvements, CPython optimizations, deprecations, and potential +porting issues. -Please read on for a comprehensive list of user-facing changes. - - -.. PEP-sized items next. -.. _pep-4XX: - -.. PEP 4XX: Virtual Environments -.. ============================= - - -.. (Implemented by Foo Bar.) - -.. .. seealso:: - - :pep:`4XX` - Python Virtual Environments - PEP written by Carl Meyer +New Features +============ +.. _whatsnew-pep-492: PEP 492 - Coroutines with async and await syntax ------------------------------------------------ -The PEP added dedicated syntax for declaring :term:`coroutines <coroutine>`, -:keyword:`await` expressions, new asynchronous :keyword:`async for` -and :keyword:`async with` statements. - -Example:: - - async def read_data(db): - async with db.transaction(): - data = await db.fetch('SELECT ...') - -PEP written and implemented by Yury Selivanov. +:pep:`492` greatly improves support for asynchronous programming in Python +by adding :term:`awaitable objects <awaitable>`, +:term:`coroutine functions <coroutine function>`, +:term:`asynchronous iteration <asynchronous iterable>`, +and :term:`asynchronous context managers <asynchronous context manager>`. + +Coroutine functions are declared using the new :keyword:`async def` syntax:: + + >>> async def coro(): + ... return 'spam' + +Inside a coroutine function, a new :keyword:`await` expression can be used +to suspend coroutine execution until the result is available. Any object +can be *awaited*, as long as it implements the :term:`awaitable` protocol by +defining the :meth:`__await__` method. + +PEP 492 also adds :keyword:`async for` statement for convenient iteration +over asynchronous iterables. + +An example of a simple HTTP client written using the new syntax:: + + import asyncio + + async def http_get(domain): + reader, writer = await asyncio.open_connection(domain, 80) + + writer.write(b'\r\n'.join([ + b'GET / HTTP/1.1', + b'Host: %b' % domain.encode('latin-1'), + b'Connection: close', + b'', b'' + ])) + + async for line in reader: + print('>>>', line) + + writer.close() + + loop = asyncio.get_event_loop() + try: + loop.run_until_complete(http_get('example.com')) + finally: + loop.close() + + +Similarly to asynchronous iteration, there is a new syntax for asynchronous +context managers:: + + >>> import asyncio + >>> async def coro1(lock): + ... print('coro1: waiting for lock') + ... async with lock: + ... print('coro1: holding the lock') + ... await asyncio.sleep(1) + ... print('coro1: releasing the lock') + ... + >>> async def coro2(lock): + ... print('coro2: waiting for lock') + ... async with lock: + ... print('coro2: holding the lock') + ... await asyncio.sleep(1) + ... print('coro2: releasing the lock') + ... + >>> loop = asyncio.get_event_loop() + >>> lock = asyncio.Lock() + >>> coros = asyncio.gather(coro1(lock), coro2(lock), loop=loop) + >>> loop.run_until_complete(coros) + coro1: waiting for lock + coro1: holding the lock + coro2: waiting for lock + coro1: releasing the lock + coro2: holding the lock + coro2: releasing the lock + >>> loop.close() + +Note that both :keyword:`async for` and :keyword:`async with` can only +be used inside a coroutine function declared with :keyword:`async def`. + +Coroutine functions are intended to be ran inside a compatible event loop, +such as :class:`asyncio.Loop`. .. seealso:: :pep:`492` -- Coroutines with async and await syntax + PEP written and implemented by Yury Selivanov. -PEP 461 - Formatting support for bytes and bytearray ----------------------------------------------------- +PEP 465 - A dedicated infix operator for matrix multiplication +-------------------------------------------------------------- -This PEP proposes adding % formatting operations similar to Python 2's ``str`` -type to :class:`bytes` and :class:`bytearray`. +:pep:`465` adds the ``@`` infix operator for matrix multiplication. +Currently, no builtin Python types implement the new operator, however, it +can be implemented by defining :meth:`__matmul__`, :meth:`__rmatmul__`, +and :meth:`__imatmul__` for regular, reflected, and in-place matrix +multiplication. The semantics of these methods is similar to that of +methods defining other infix arithmetic operators. -Examples:: +Matrix multiplication is a notably common operation in many fields of +mathematics, science, engineering, and the addition of ``@`` allows writing +cleaner code:: - >>> b'Hello %s!' % b'World' - b'Hello World!' - >>> b'x=%i y=%f' % (1, 2.5) - b'x=1 y=2.500000' + S = (H @ beta - r).T @ inv(H @ V @ H.T) @ (H @ beta - r) -Unicode is not allowed for ``%s``, but it is accepted by ``%a`` (equivalent of -``repr(obj).encode('ascii', 'backslashreplace')``):: +instead of:: - >>> b'Hello %s!' % 'World' - Traceback (most recent call last): - File "<stdin>", line 1, in <module> - TypeError: %b requires bytes, or an object that implements __bytes__, not 'str' - >>> b'price: %a' % '10€' - b"price: '10\\u20ac'" + S = dot((dot(H, beta) - r).T, + dot(inv(dot(dot(H, V), H.T)), dot(H, beta) - r)) -.. seealso:: +An upcoming release of NumPy 1.10 will add support for the new operator:: - :pep:`461` -- Adding % formatting to bytes and bytearray + >>> import numpy + >>> x = numpy.ones(3) + >>> x + array([ 1., 1., 1.]) -PEP 465 - A dedicated infix operator for matrix multiplication --------------------------------------------------------------- + >>> m = numpy.eye(3) + >>> m + array([[ 1., 0., 0.], + [ 0., 1., 0.], + [ 0., 0., 1.]]) + + >>> x @ m + array([ 1., 1., 1.]) -This PEP proposes a new binary operator to be used for matrix multiplication, -called ``@``. (Mnemonic: ``@`` is ``*`` for mATrices.) .. seealso:: :pep:`465` -- A dedicated infix operator for matrix multiplication + PEP written by Nathaniel J. Smith; implemented by Benjamin Peterson. PEP 448 - Additional Unpacking Generalizations ---------------------------------------------- -This PEP proposes extended usages of the ``*`` iterable unpacking -operator and ``**`` dictionary unpacking operators -to allow unpacking in more positions, an arbitrary number of -times, and in additional circumstances. Specifically, -in function calls, in comprehensions and generator expressions, and -in displays. +:pep:`448` extends the allowed uses of the ``*`` iterable unpacking +operator and ``**`` dictionary unpacking operator. It is now possible +to use an arbitrary number of unpackings in function calls:: + + >>> print(*[1], *[2], 3, *[4, 5]) + 1 2 3 4 5 -Function calls are proposed to support an arbitrary number of -unpackings rather than just one:: + >>> def fn(a, b, c, d): + ... print(a, b, c, d) + ... - >>> print(*[1], *[2], 3) - 1 2 3 - >>> dict(**{'x': 1}, y=2, **{'z': 3}) - {'x': 1, 'y': 2, 'z': 3} + >>> fn(**{'a': 1, 'c': 3}, **{'b': 2, 'd': 4}) + 1 2 3 4 -Unpacking is proposed to be allowed inside tuple, list, set, -and dictionary displays:: +Similarly, tuple, list, set, and dictionary displays allow multiple +unpackings:: >>> *range(4), 4 (0, 1, 2, 3, 4) >>> [*range(4), 4] [0, 1, 2, 3, 4] - >>> {*range(4), 4} - {0, 1, 2, 3, 4} + >>> {*range(4), 4, *(5, 6, 7)} + {0, 1, 2, 3, 4, 5, 6, 7} >>> {'x': 1, **{'y': 2}} {'x': 1, 'y': 2} -In dictionaries, later values will always override earlier ones:: +.. seealso:: + + :pep:`448` -- Additional Unpacking Generalizations + PEP written by Joshua Landau; implemented by Neil Girdhar, + Thomas Wouters, and Joshua Landau. - >>> {'x': 1, **{'x': 2}} - {'x': 2} - >>> {**{'x': 2}, 'x': 1} - {'x': 1} +PEP 461 - % formatting support for bytes and bytearray +------------------------------------------------------ + +PEP 461 adds % formatting to :class:`bytes` and :class:`bytearray`, aiding in +handling data that is a mixture of binary and ASCII compatible text. This +feature also eases porting such code from Python 2. + +Examples:: + + >>> b'Hello %s!' % b'World' + b'Hello World!' + >>> b'x=%i y=%f' % (1, 2.5) + b'x=1 y=2.500000' + +Unicode is not allowed for ``%s``, but it is accepted by ``%a`` (equivalent of +``repr(obj).encode('ascii', 'backslashreplace')``):: + + >>> b'Hello %s!' % 'World' + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + TypeError: %b requires bytes, or an object that implements __bytes__, not 'str' + >>> b'price: %a' % '10€' + b"price: '10\\u20ac'" .. seealso:: - :pep:`448` -- Additional Unpacking Generalizations + :pep:`461` -- Adding % formatting to bytes and bytearray + PEP written by Ethan Furman; implemented by Neil Schemenauer and + Ethan Furman. PEP 484 - Type Hints @@ -261,8 +374,8 @@ where annotations are not available. For example, here is a simple function whose argument and return type are declared in the annotations:: - def greeting(name: str) -> str: - return 'Hello ' + name + def greeting(name: str) -> str: + return 'Hello ' + name The type system supports unions, generic types, and a special type named ``Any`` which is consistent with (i.e. assignable to and from) all @@ -270,8 +383,10 @@ types. .. seealso:: - * :pep:`484` -- Type Hints * :mod:`typing` module documentation + * :pep:`484` -- Type Hints + PEP written by Guido van Rossum, Jukka Lehtosalo, and Łukasz Langa; + implemented by Guido van Rossum. PEP 471 - os.scandir() function -- a better and faster directory iterator @@ -282,12 +397,10 @@ to the standard library. Additionally, :func:`os.walk` is now implemented using :func:`os.scandir`, which speeds it up by 3-5 times on POSIX systems and by 7-20 times on Windows systems. -PEP and implementation written by Ben Hoyt with the help of Victor Stinner. - .. seealso:: - :pep:`471` -- os.scandir() function -- a better and faster directory - iterator + :pep:`471` -- os.scandir() function -- a better and faster directory iterator + PEP written and implemented by Ben Hoyt with the help of Victor Stinner. PEP 475: Retry system calls failing with EINTR @@ -302,67 +415,44 @@ Examples of functions which are now retried when interrupted by a signal instead of raising :exc:`InterruptedError` if the Python signal handler does not raise an exception: -* :func:`open`, :func:`os.open`, :func:`io.open` -* functions of the :mod:`faulthandler` module -* :mod:`os` functions: - - - :func:`os.fchdir` - - :func:`os.fchmod` - - :func:`os.fchown` - - :func:`os.fdatasync` - - :func:`os.fstat` - - :func:`os.fstatvfs` - - :func:`os.fsync` - - :func:`os.ftruncate` - - :func:`os.mkfifo` - - :func:`os.mknod` - - :func:`os.posix_fadvise` - - :func:`os.posix_fallocate` - - :func:`os.pread` - - :func:`os.pwrite` - - :func:`os.read` - - :func:`os.readv` - - :func:`os.sendfile` - - :func:`os.wait3` - - :func:`os.wait4` - - :func:`os.wait` - - :func:`os.waitid` - - :func:`os.waitpid` - - :func:`os.write` - - :func:`os.writev` - - special cases: :func:`os.close` and :func:`os.dup2` now ignore - :py:data:`~errno.EINTR` error, the syscall is not retried (see the PEP - for the rationale) - -* :mod:`select` functions: - - - :func:`select.devpoll.poll` - - :func:`select.epoll.poll` - - :func:`select.kqueue.control` - - :func:`select.poll.poll` - - :func:`select.select` - -* :func:`socket.socket` methods: - - - :meth:`~socket.socket.accept` - - :meth:`~socket.socket.connect` (except for non-blocking sockets) - - :meth:`~socket.socket.recv` - - :meth:`~socket.socket.recvfrom` - - :meth:`~socket.socket.recvmsg` - - :meth:`~socket.socket.send` - - :meth:`~socket.socket.sendall` - - :meth:`~socket.socket.sendmsg` - - :meth:`~socket.socket.sendto` - -* :func:`signal.sigtimedwait`, :func:`signal.sigwaitinfo` -* :func:`time.sleep` - -PEP and implementation written by Charles-François Natali and Victor Stinner, -with the help of Antoine Pitrou (the french connection). +* :func:`open`, :func:`os.open`, :func:`io.open`; + +* functions of the :mod:`faulthandler` module; + +* :mod:`os` functions: :func:`~os.fchdir`, :func:`~os.fchmod`, + :func:`~os.fchown`, :func:`~os.fdatasync`, :func:`~os.fstat`, + :func:`~os.fstatvfs`, :func:`~os.fsync`, :func:`~os.ftruncate`, + :func:`~os.mkfifo`, :func:`~os.mknod`, :func:`~os.posix_fadvise`, + :func:`~os.posix_fallocate`, :func:`~os.pread`, :func:`~os.pwrite`, + :func:`~os.read`, :func:`~os.readv`, :func:`~os.sendfile`, + :func:`~os.wait3`, :func:`~os.wait4`, :func:`~os.wait`, + :func:`~os.waitid`, :func:`~os.waitpid`, :func:`~os.write`, + :func:`~os.writev`; + +* special cases: :func:`os.close` and :func:`os.dup2` now ignore + :py:data:`~errno.EINTR` error, the syscall is not retried (see the PEP + for the rationale); + +* :mod:`select` functions: :func:`~select.devpoll.poll`, + :func:`~select.epoll.poll`, :func:`~select.kqueue.control`, + :func:`~select.poll.poll`, :func:`~select.select`; + +* :func:`socket.socket` methods: :meth:`~socket.socket.accept`, + :meth:`~socket.socket.connect` (except for non-blocking sockets), + :meth:`~socket.socket.recv`, :meth:`~socket.socket.recvfrom`, + :meth:`~socket.socket.recvmsg`, :meth:`~socket.socket.send`, + :meth:`~socket.socket.sendall`, :meth:`~socket.socket.sendmsg`, + :meth:`~socket.socket.sendto`; + +* :func:`signal.sigtimedwait`, :func:`signal.sigwaitinfo`; + +* :func:`time.sleep`. .. seealso:: :pep:`475` -- Retry system calls failing with EINTR + PEP and implementation written by Charles-François Natali and + Victor Stinner, with the help of Antoine Pitrou (the french connection). PEP 479: Change StopIteration handling inside generators @@ -378,12 +468,11 @@ be used:: Without a ``__future__`` import, a :exc:`PendingDeprecationWarning` will be raised. -PEP written by Chris Angelico and Guido van Rossum. Implemented by -Chris Angelico, Yury Selivanov and Nick Coghlan. - .. seealso:: :pep:`479` -- Change StopIteration handling inside generators + PEP written by Chris Angelico and Guido van Rossum. Implemented by + Chris Angelico, Yury Selivanov and Nick Coghlan. PEP 486: Make the Python Launcher aware of virtual environments @@ -397,6 +486,7 @@ environment will be used. .. seealso:: :pep:`486` -- Make the Python Launcher aware of virtual environments + PEP written and implemented by Paul Moore. PEP 488: Elimination of PYO files @@ -414,6 +504,7 @@ has an updated API to help with this change. .. seealso:: :pep:`488` -- Elimination of PYO files + PEP written and implemented by Brett Cannon. PEP 489: Multi-phase extension module initialization @@ -429,7 +520,10 @@ rather than being restricted to ASCII. .. seealso:: - :pep:`488` -- Multi-phase extension module initialization + :pep:`489` -- Multi-phase extension module initialization + PEP written by Petr Viktorin, Stefan Behnel, and Nick Coghlan; + implemented by Petr Viktorin. + PEP 485: A function for testing approximate equality ---------------------------------------------------- @@ -442,6 +536,9 @@ close is determined according to given absolute and relative tolerances. .. seealso:: :pep:`485` -- A function for testing approximate equality + PEP written by Christopher Barker; implemented by Chris Barker and + Tal Einat. + Other Language Changes ====================== @@ -465,6 +562,9 @@ Some smaller changes made to the core Python language are: * New Tajik :ref:`codec <standard-encodings>` ``koi8_t``. (Contributed by Serhiy Storchaka in :issue:`22681`.) +* Circular imports involving relative imports are now supported. + (Contributed by Brett Cannon and Antoine Pitrou in :issue:`17636`.) + New Modules =========== @@ -496,7 +596,15 @@ argparse * :class:`~argparse.ArgumentParser` now allows to disable :ref:`abbreviated usage <prefix-matching>` of long options by setting :ref:`allow_abbrev` to ``False``. - (Contributed by Jonathan Paugh, Steven Bethard, paul j3 and Daniel Eriksson.) + (Contributed by Jonathan Paugh, Steven Bethard, paul j3 and Daniel Eriksson + in :issue:`14910`.) + +bz2 +--- + +* New option *max_length* for :meth:`~bz2.BZ2Decompressor.decompress` + to limit the maximum size of decompressed data. + (Contributed by Nikolaus Rath in :issue:`15955`.) cgi --- @@ -530,6 +638,33 @@ collections (Contributed by Berker Peksag in :issue:`24064`.) +* :class:`~collections.deque` has new methods: + :meth:`~collections.deque.index`, :meth:`~collections.deque.insert`, and + :meth:`~collections.deque.copy`. This allows deques to be registered as a + :class:`~collections.abc.MutableSequence` and improves their + substitutablity for lists. + (Contributed by Raymond Hettinger :issue:`23704`.) + +* :class:`~collections.UserString` now supports ``__getnewargs__``, + ``__rmod__``, ``casefold``, ``format_map``, ``isprintable``, and + ``maketrans`` methods. + (Contributed by Joe Jevnik in :issue:`22189`.) + +* :class:`collections.OrderedDict` is now implemented in C, which improves + its performance between 4x to 100x times. + (Contributed by Eric Snow in :issue:`16991`.) + +collections.abc +--------------- + +* New :class:`~collections.abc.Generator` abstract base class. + (Contributed by Stefan Behnel in :issue:`24018`.) + +* New :class:`~collections.abc.Coroutine`, + :class:`~collections.abc.AsyncIterator`, and + :class:`~collections.abc.AsyncIterable` abstract base classes. + (Contributed by Yury Selivanov in :issue:`24184`.) + compileall ---------- @@ -537,6 +672,20 @@ compileall can now do parallel bytecode compilation. (Contributed by Claudiu Popa in :issue:`16104`.) +* *quiet* parameter of :func:`compileall.compile_dir`, + :func:`compileall.compile_file`, and :func:`compileall.compile_path` + functions now has a multilevel value. New ``-qq`` command line option + is available for suppressing the output. + (Contributed by Thomas Kluyver in :issue:`21338`.) + +concurrent.futures +------------------ + +* :meth:`~concurrent.futures.Executor.map` now takes a *chunksize* + argument to allow batching of tasks in child processes and improve + performance of ProcessPoolExecutor. + (Contributed by Dan O'Reilly in :issue:`11271`.) + contextlib ---------- @@ -560,7 +709,8 @@ difflib (Contributed by Berker Peksag in :issue:`2052`.) * It's now possible to compare lists of byte strings with - :func:`difflib.diff_bytes` (fixes a regression from Python 2). + :func:`difflib.diff_bytes`. This fixes a regression from Python 2. + (Contributed by Terry J. Reedy and Greg Ward in :issue:`17445`.) distutils --------- @@ -599,13 +749,36 @@ email :rfc:`6532` and used with an SMTP server that supports the :rfc:`6531` ``SMTPUTF8`` extension. (Contributed by R. David Murray in :issue:`24211`.) +faulthandler +------------ + +* :func:`~faulthandler.enable`, :func:`~faulthandler.register`, + :func:`~faulthandler.dump_traceback` and + :func:`~faulthandler.dump_traceback_later` functions now accept file + descriptors. (Contributed by Wei Wu in :issue:`23566`.) + +functools +--------- + +* Most of :func:`~functools.lru_cache` machinery is now implemented in C. + (Contributed by Matt Joiner, Alexey Kachayev, and Serhiy Storchaka + in :issue:`14373`.) + glob ---- +00002-5938667 + * :func:`~glob.iglob` and :func:`~glob.glob` now support recursive search in subdirectories using the "``**``" pattern. (Contributed by Serhiy Storchaka in :issue:`13968`.) +heapq +----- + +* :func:`~heapq.merge` has two new optional parameters ``reverse`` and + ``key``. (Contributed by Raymond Hettinger in :issue:`13742`.) + idlelib and IDLE ---------------- @@ -638,7 +811,9 @@ imghdr ------ * :func:`~imghdr.what` now recognizes the `OpenEXR <http://www.openexr.com>`_ - format. (Contributed by Martin Vignali and Claudiu Popa in :issue:`20295`.) + format (contributed by Martin Vignali and Claudiu Popa in :issue:`20295`), + and the `WebP <https://en.wikipedia.org/wiki/WebP>`_ format (contributed + by Fabrice Aneche and Claudiu Popa in :issue:`20197`.) importlib --------- @@ -655,6 +830,7 @@ importlib * :func:`importlib.util.module_from_spec` is now the preferred way to create a new module. Compared to :class:`types.ModuleType`, this new function will set the various import-controlled attributes based on the passed-in spec object. + (Contributed by Brett Cannon in :issue:`20383`.) inspect ------- @@ -680,6 +856,17 @@ inspect * New :func:`~inspect.getcoroutinelocals` and :func:`~inspect.getcoroutinestate` functions. (Contributed by Yury Selivanov in :issue:`24400`.) +* :func:`~inspect.stack`, :func:`~inspect.trace`, :func:`~inspect.getouterframes`, + and :func:`~inspect.getinnerframes` now return a list of named tuples. + (Contributed by Daniel Shahaf in :issue:`16808`.) + +io +-- + +* New Python implementation of :class:`io.FileIO` to make dependency on + ``_io`` module optional. + (Contributed by Serhiy Storchaka in :issue:`21859`.) + ipaddress --------- @@ -699,14 +886,58 @@ json * JSON decoder now raises :exc:`json.JSONDecodeError` instead of :exc:`ValueError`. (Contributed by Serhiy Storchaka in :issue:`19361`.) +locale +------ + + * New :func:`~locale.delocalize` function to convert a string into a + normalized number string, following the ``LC_NUMERIC`` settings. + (Contributed by Cédric Krier in :issue:`13918`.) + +logging +------- + +* All logging methods :meth:`~logging.Logger.log`, :meth:`~logging.Logger.exception`, + :meth:`~logging.Logger.critical`, :meth:`~logging.Logger.debug`, etc, + now accept exception instances for ``exc_info`` parameter, in addition + to boolean values and exception tuples. + (Contributed by Yury Selivanov in :issue:`20537`.) + +* :class:`~logging.handlers.HTTPHandler` now accepts optional + :class:`ssl.SSLContext` instance to configure the SSL settings used + for HTTP connection. + (Contributed by Alex Gaynor in :issue:`22788`.) + +* :class:`~logging.handlers.QueueListener` now takes a *respect_handler_level* + keyword argument which, if set to ``True``, will pass messages to handlers + taking handler levels into account. (Contributed by Vinay Sajip.) + +lzma +---- + +* New option *max_length* for :meth:`~lzma.LZMADecompressor.decompress` + to limit the maximum size of decompressed data. + (Contributed by Martin Panter in :issue:`15955`.) + math ---- * :data:`math.inf` and :data:`math.nan` constants added. (Contributed by Mark Dickinson in :issue:`23185`.) -* :func:`math.isclose` function added. + +* New :func:`~math.isclose` function. (Contributed by Chris Barker and Tal Einat in :issue:`24270`.) +* New :func:`~math.gcd` function. The :func:`fractions.gcd` function now is + deprecated. + (Contributed by Mark Dickinson and Serhiy Storchaka in :issue:`22486`.) + +operator +-------- + +* :func:`~operator.attrgetter`, :func:`~operator.itemgetter`, and + :func:`~operator.methodcaller` objects now support pickling. + (Contributed by Josh Rosenberg and Serhiy Storchaka in :issue:`22955`.) + os -- @@ -723,6 +954,14 @@ os now used when available. On OpenBSD 5.6 and newer, the C ``getentropy()`` function is now used. These functions avoid the usage of an internal file descriptor. + (Contributed by Victor Stinner in :issue:`22181`.) + +* New :func:`os.get_blocking` and :func:`os.set_blocking` functions to + get and set the blocking mode of file descriptors. + (Contributed by Victor Stinner in :issue:`22054`.) + +* :func:`~os.truncate` and :func:`~os.ftruncate` are now supported on + Windows. (Contributed by Steve Dower in :issue:`23668`.) os.path ------- @@ -731,6 +970,25 @@ os.path Unlike the :func:`~os.path.commonprefix` function, it always returns a valid path. (Contributed by Rafik Draoui and Serhiy Storchaka in :issue:`10395`.) +pathlib +------- + +* New :meth:`~pathlib.Path.samefile` method to check if other path object + points to the same file. (Contributed by Vajrasky Kok and Antoine Pitrou + in :issue:`19775`.) + +* :meth:`~pathlib.Path.mkdir` has a new optional parameter ``exist_ok`` + to mimic ``mkdir -p`` and :func:`os.makrdirs` functionality. + (Contributed by Berker Peksag in :issue:`21539`.) + +* New :meth:`~pathlib.Path.expanduser` to expand ``~`` and ``~user`` + constructs. + (Contributed by Serhiy Storchaka and Claudiu Popa in :issue:`19776`.) + +* New class method :meth:`~pathlib.Path.home` to get an instance of + :class:`~pathlib.Path` object representing the user’s home directory. + (Contributed by Victor Salgado and Mayank Tripathi in :issue:`19777`.) + pickle ------ @@ -754,6 +1012,12 @@ re * Now unmatched groups are replaced with empty strings in :func:`re.sub` and :func:`re.subn`. (Contributed by Serhiy Storchaka in :issue:`1519638`.) +readline +-------- + +* New :func:`~readline.append_history_file` function. + (Contributed by Bruno Cauet in :issue:`22940`.) + shutil ------ @@ -762,6 +1026,9 @@ shutil :func:`~shutil.copy2` if there is a need to ignore metadata. (Contributed by Claudiu Popa in :issue:`19840`.) +* :func:`~shutil.make_archive` now supports *xztar* format. + (Contributed by Serhiy Storchaka in :issue:`5411`.) + signal ------ @@ -840,6 +1107,17 @@ ssl :meth:`SSLContext.wrap_bio <ssl.SSLContext.wrap_bio>` method. (Contributed by Geert Jansen in :issue:`21965`.) +* New :meth:`~ssl.SSLSocket.version` to query the actual protocol version + in use. (Contributed by Antoine Pitrou in :issue:`20421`.) + +* New :meth:`~ssl.SSLObject.shared_ciphers` and + :meth:`~ssl.SSLSocket.shared_ciphers` methods to fetch the client's + list of ciphers sent at handshake. + (Contributed by Benjamin Peterson in :issue:`23186`.) + +* :func:`~ssl.match_hostname` now supports matching of IP addresses. + (Contributed by Antoine Pitrou in :issue:`23239`.) + socket ------ @@ -853,6 +1131,9 @@ socket anymore each time bytes are received or sent. The socket timeout is now the maximum total duration to send all data. +* Functions with timeouts now use a monotonic clock, instead of a + system clock. (Contributed by Victor Stinner in :issue:`22043`.) + subprocess ---------- @@ -867,6 +1148,9 @@ sys * New :func:`~sys.set_coroutine_wrapper` and :func:`~sys.get_coroutine_wrapper` functions. (Contributed by Yury Selivanov in :issue:`24017`.) +* New :func:`~sys.is_finalizing` to check for :term:`interpreter shutdown`. + (Contributed by Antoine Pitrou in :issue:`22696`.) + sysconfig --------- @@ -883,15 +1167,29 @@ tarfile methods now take a keyword parameter *numeric_only*. If set to ``True``, the extracted files and directories will be owned by the numeric uid and gid from the tarfile. If set to ``False`` (the default, and the behavior in - versions prior to 3.5), they will be owned bythe named user and group in the + versions prior to 3.5), they will be owned by the named user and group in the tarfile. (Contributed by Michael Vogt and Eric Smith in :issue:`23193`.) +threading +--------- + +* :meth:`~threading.Lock.acquire` and :meth:`~threading.RLock.acquire` + now use a monotonic clock for managing timeouts. + (Contributed by Victor Stinner in :issue:`22043`.) + time ---- * The :func:`time.monotonic` function is now always available. (Contributed by Victor Stinner in :issue:`22043`.) +timeit +------ + +* New command line option ``-u`` or ``--unit=U`` to specify a time unit for + the timer output. Supported options are ``usec``, ``msec``, or ``sec``. + (Contributed by Julian Gindi in :issue:`18983`.) + tkinter ------- @@ -900,6 +1198,22 @@ tkinter module which makes no permanent changes to environment variables. (Contributed by Zachary Ware in :issue:`20035`.) +traceback +--------- + +* New :func:`~traceback.walk_stack` and :func:`~traceback.walk_tb` + functions to conveniently traverse frame and traceback objects. + (Contributed by Robert Collins in :issue:`17911`.) + +* New lightweight classes: :class:`~traceback.TracebackException`, + :class:`~traceback.StackSummary`, and :class:`traceback.FrameSummary`. + (Contributed by Robert Collins in :issue:`17911`.) + +* :func:`~traceback.print_tb` and :func:`~traceback.print_stack` now support + negative values for the *limit* argument. + (Contributed by Dmitry Kazakov in :issue:`22619`.) + + types ----- @@ -924,12 +1238,22 @@ urllib control the encoding of query parts if needed. (Contributed by Samwyse and Arnon Yaari in :issue:`13866`.) +* :func:`~urllib.request.urlopen` accepts an :class:`ssl.SSLContext` + object as a *context* argument, which will be used for the HTTPS + connection. (Contributed by Alex Gaynor in :issue:`22366`.) + unicodedata ----------- * The :mod:`unicodedata` module now uses data from `Unicode 8.0.0 <http://unicode.org/versions/Unicode8.0.0/>`_. +unittest +-------- + +* New command line option ``--locals`` to show local variables in + tracebacks. + (Contributed by Robert Collins in :issue:`22936`.) wsgiref ------- @@ -943,6 +1267,10 @@ xmlrpc * :class:`xmlrpc.client.ServerProxy` is now a :term:`context manager`. (Contributed by Claudiu Popa in :issue:`20627`.) +* :class:`~xmlrpc.client.ServerProxy` constructor now accepts an optional + :class:`ssl.SSLContext` instance. + (Contributed by Alex Gaynor in :issue:`22960`.) + xml.sax ------- @@ -950,14 +1278,6 @@ xml.sax :class:`~xml.sax.xmlreader.InputSource` object. (Contributed by Serhiy Storchaka in :issue:`2175`.) -faulthandler ------------- - -* :func:`~faulthandler.enable`, :func:`~faulthandler.register`, - :func:`~faulthandler.dump_traceback` and - :func:`~faulthandler.dump_traceback_later` functions now accept file - descriptors. (Contributed by Wei Wu in :issue:`23566`.) - zipfile ------- @@ -968,6 +1288,14 @@ zipfile creation) mode. (Contributed by Serhiy Storchaka in :issue:`21717`.) +Other module-level changes +========================== + +* Many functions in modules :mod:`mmap`, :mod:`ossaudiodev`, :mod:`socket`, + :mod:`ssl`, and :mod:`codecs`, now accept writable bytes-like objects. + (Contributed by Serhiy Storchaka in :issue:`23001`.) + + Optimizations ============= @@ -988,7 +1316,10 @@ The following performance enhancements have been added: :meth:`~ipaddress.IPv4Network.subnets`, :meth:`~ipaddress.IPv4Network.supernet`, :func:`~ipaddress.summarize_address_range`, :func:`~ipaddress.collapse_addresses`. The speed up can range from 3x to 15x. - (:issue:`21486`, :issue:`21487`, :issue:`20826`) + (See :issue:`21486`, :issue:`21487`, :issue:`20826`, :issue:`23266`.) + +* Pickling of :mod:`ipaddress` classes was optimized to produce significantly + smaller output. (Contributed by Serhiy Storchaka in :issue:`23133`.) * Many operations on :class:`io.BytesIO` are now 50% to 100% faster. (Contributed by Serhiy Storchaka in :issue:`15381` and David Wilson in @@ -1001,6 +1332,29 @@ The following performance enhancements have been added: * The UTF-32 encoder is now 3x to 7x faster. (Contributed by Serhiy Storchaka in :issue:`15027`.) +* Regular expressions are now parsed up to 10% faster. + (Contributed by Serhiy Storchaka in :issue:`19380`.) + +* :func:`json.dumps` was optimized to run with ``ensure_ascii=False`` + as fast as with ``ensure_ascii=True``. + (Contributed by Naoki Inada in :issue:`23206`.) + +* :c:func:`PyObject_IsInstance` and :c:func:`PyObject_IsSubclass` have + been sped up in the common case that the second argument has metaclass + :class:`type`. + (Contributed Georg Brandl by in :issue:`22540`.) + +* Method caching was slightly improved, yielding up to 5% performance + improvement in some benchmarks. + (Contributed by Antoine Pitrou in :issue:`22847`.) + +* Objects from :mod:`random` module now use 2x less memory on 64-bit + builds. + (Contributed by Serhiy Storchaka in :issue:`23488`.) + +* property() getter calls are up to 25% faster. + (Contributed by Joe Jevnik in :issue:`23910`.) + Build and C API Changes ======================= @@ -1040,6 +1394,9 @@ Deprecated Python modules, functions and methods * The :mod:`formatter` module has now graduated to full deprecation and is still slated for removal in Python 3.6. +* :func:`~asyncio.async` was deprecated in favour of + :func:`~asyncio.ensure_future`. + * :mod:`smtpd` has in the past always decoded the DATA portion of email messages using the ``utf-8`` codec. This can now be controlled by the new *decode_data* keyword to :class:`~smtpd.SMTPServer`. The default value is @@ -1077,6 +1434,9 @@ Deprecated Python modules, functions and methods deprecated in favor of :func:`inspect.signature` API. (See :issue:`20438` for details.) +* Use of ``re.LOCALE`` flag with str patterns or ``re.ASCII`` is now + deprecated. (Contributed by Serhiy Storchaka in :issue:`22407`.) + Deprecated functions and types of the C API ------------------------------------------- @@ -1084,12 +1444,6 @@ Deprecated functions and types of the C API * None yet. -Deprecated features -------------------- - -* None yet. - - Removed ======= @@ -1247,3 +1601,5 @@ Changes in the C API :c:type:`PyTypeObject` was replaced with a :c:member:`tp_as_async` slot. Refer to :ref:`coro-objects` for new types, structures and functions. + +* :c:member:`PyTypeObject.tp_finalize` is now part of stable ABI. diff --git a/Include/object.h b/Include/object.h index 8afcbe9..4d286ef 100644 --- a/Include/object.h +++ b/Include/object.h @@ -351,7 +351,8 @@ typedef struct _typeobject { printfunc tp_print; getattrfunc tp_getattr; setattrfunc tp_setattr; - PyAsyncMethods *tp_as_async; /* formerly known as tp_compare or tp_reserved */ + PyAsyncMethods *tp_as_async; /* formerly known as tp_compare (Python 2) + or tp_reserved (Python 3) */ reprfunc tp_repr; /* Method suites for standard classes */ diff --git a/Lib/_pyio.py b/Lib/_pyio.py index 50ad9ff..f47df91 100644 --- a/Lib/_pyio.py +++ b/Lib/_pyio.py @@ -8,12 +8,13 @@ import codecs import errno import array import stat +import sys # Import _thread instead of threading to reduce startup cost try: from _thread import allocate_lock as Lock except ImportError: from _dummy_thread import allocate_lock as Lock -if os.name == 'win32': +if sys.platform in {'win32', 'cygwin'}: from msvcrt import setmode as _setmode else: _setmode = None @@ -720,6 +720,11 @@ class FieldStorage: self.bytes_read += len(hdr_text) parser.feed(hdr_text.decode(self.encoding, self.errors)) headers = parser.close() + + # Some clients add Content-Length for part headers, ignore them + if 'content-length' in headers: + del headers['content-length'] + part = klass(self.fp, headers, ib, environ, keep_blank_values, strict_parsing,self.limit-self.bytes_read, self.encoding, self.errors) diff --git a/Lib/collections/__init__.py b/Lib/collections/__init__.py index 80dc4f6..943cbca 100644 --- a/Lib/collections/__init__.py +++ b/Lib/collections/__init__.py @@ -320,23 +320,14 @@ class {typename}(tuple): 'Return a nicely formatted representation string' return self.__class__.__name__ + '({repr_fmt})' % self - @property - def __dict__(self): - 'A new OrderedDict mapping field names to their values' - return OrderedDict(zip(self._fields, self)) - def _asdict(self): 'Return a new OrderedDict which maps field names to their values.' - return self.__dict__ + return OrderedDict(zip(self._fields, self)) def __getnewargs__(self): 'Return self as a plain tuple. Used by copy and pickle.' return tuple(self) - def __getstate__(self): - 'Exclude the OrderedDict from pickling' - return None - {field_defs} """ @@ -901,9 +892,8 @@ class ChainMap(MutableMapping): __copy__ = copy def new_child(self, m=None): # like Django's Context.push() - ''' - New ChainMap with a new map followed by all previous maps. If no - map is provided, an empty dict is used. + '''New ChainMap with a new map followed by all previous maps. + If no map is provided, an empty dict is used. ''' if m is None: m = {} diff --git a/Lib/configparser.py b/Lib/configparser.py index ecd0660..3a9fb56 100644 --- a/Lib/configparser.py +++ b/Lib/configparser.py @@ -263,12 +263,9 @@ class InterpolationMissingOptionError(InterpolationError): """A string substitution required a setting which was not available.""" def __init__(self, option, section, rawval, reference): - msg = ("Bad value substitution:\n" - "\tsection: [%s]\n" - "\toption : %s\n" - "\tkey : %s\n" - "\trawval : %s\n" - % (section, option, reference, rawval)) + msg = ("Bad value substitution: option {!r} in section {!r} contains " + "an interpolation key {!r} which is not a valid option name. " + "Raw value: {!r}".format(option, section, reference, rawval)) InterpolationError.__init__(self, option, section, msg) self.reference = reference self.args = (option, section, rawval, reference) @@ -286,11 +283,11 @@ class InterpolationDepthError(InterpolationError): """Raised when substitutions are nested too deeply.""" def __init__(self, option, section, rawval): - msg = ("Value interpolation too deeply recursive:\n" - "\tsection: [%s]\n" - "\toption : %s\n" - "\trawval : %s\n" - % (section, option, rawval)) + msg = ("Recursion limit exceeded in value substitution: option {!r} " + "in section {!r} contains an interpolation key which " + "cannot be substituted in {} steps. Raw value: {!r}" + "".format(option, section, MAX_INTERPOLATION_DEPTH, + rawval)) InterpolationError.__init__(self, option, section, msg) self.args = (option, section, rawval) @@ -406,8 +403,9 @@ class BasicInterpolation(Interpolation): def _interpolate_some(self, parser, option, accum, rest, section, map, depth): + rawval = parser.get(section, option, raw=True, fallback=rest) if depth > MAX_INTERPOLATION_DEPTH: - raise InterpolationDepthError(option, section, rest) + raise InterpolationDepthError(option, section, rawval) while rest: p = rest.find("%") if p < 0: @@ -432,7 +430,7 @@ class BasicInterpolation(Interpolation): v = map[var] except KeyError: raise InterpolationMissingOptionError( - option, section, rest, var) from None + option, section, rawval, var) from None if "%" in v: self._interpolate_some(parser, option, accum, v, section, map, depth + 1) @@ -466,8 +464,9 @@ class ExtendedInterpolation(Interpolation): def _interpolate_some(self, parser, option, accum, rest, section, map, depth): + rawval = parser.get(section, option, raw=True, fallback=rest) if depth > MAX_INTERPOLATION_DEPTH: - raise InterpolationDepthError(option, section, rest) + raise InterpolationDepthError(option, section, rawval) while rest: p = rest.find("$") if p < 0: @@ -504,7 +503,7 @@ class ExtendedInterpolation(Interpolation): "More than one ':' found: %r" % (rest,)) except (KeyError, NoSectionError, NoOptionError): raise InterpolationMissingOptionError( - option, section, rest, ":".join(path)) from None + option, section, rawval, ":".join(path)) from None if "$" in v: self._interpolate_some(parser, opt, accum, v, sect, dict(parser.items(sect, raw=True)), diff --git a/Lib/distutils/_msvccompiler.py b/Lib/distutils/_msvccompiler.py index 82b78a0..03a5f10 100644 --- a/Lib/distutils/_msvccompiler.py +++ b/Lib/distutils/_msvccompiler.py @@ -100,7 +100,7 @@ def _get_vc_env(plat_spec): (line.partition('=') for line in out.splitlines()) if key and value } - + if vcruntime: env['py_vcruntime_redist'] = vcruntime return env @@ -236,7 +236,7 @@ class MSVCCompiler(CCompiler) : '/nologo', '/Ox', '/W3', '/GL', '/DNDEBUG' ] self.compile_options.append('/MD' if self._vcruntime_redist else '/MT') - + self.compile_options_debug = [ '/nologo', '/Od', '/MDd', '/Zi', '/W3', '/D_DEBUG' ] diff --git a/Lib/distutils/tests/test_msvccompiler.py b/Lib/distutils/tests/test_msvccompiler.py index 874d603..c4d911f 100644 --- a/Lib/distutils/tests/test_msvccompiler.py +++ b/Lib/distutils/tests/test_msvccompiler.py @@ -77,7 +77,7 @@ class msvccompilerTestCase(support.TempdirManager, compiler.initialize() dll = compiler._vcruntime_redist self.assertTrue(os.path.isfile(dll)) - + compiler._copy_vcruntime(tempdir) self.assertFalse(os.path.isfile(os.path.join( diff --git a/Lib/functools.py b/Lib/functools.py index 09df068..06a4ff1 100644 --- a/Lib/functools.py +++ b/Lib/functools.py @@ -567,7 +567,7 @@ def _c3_merge(sequences): break # reject the current head, it appears later else: break - if not candidate: + if candidate is None: raise RuntimeError("Inconsistent hierarchy") result.append(candidate) # remove the chosen candidate diff --git a/Lib/html/parser.py b/Lib/html/parser.py index 390d4cc..43e6411 100644 --- a/Lib/html/parser.py +++ b/Lib/html/parser.py @@ -139,7 +139,15 @@ class HTMLParser(_markupbase.ParserBase): if self.convert_charrefs and not self.cdata_elem: j = rawdata.find('<', i) if j < 0: - if not end: + # if we can't find the next <, either we are at the end + # or there's more text incoming. If the latter is True, + # we can't pass the text to handle_data in case we have + # a charref cut in half at end. Try to determine if + # this is the case before proceding by looking for an + # & near the end and see if it's followed by a space or ;. + amppos = rawdata.rfind('&', max(i, n-34)) + if (amppos >= 0 and + not re.compile(r'[\s;]').search(rawdata, amppos)): break # wait till we get all the text j = n else: diff --git a/Lib/http/server.py b/Lib/http/server.py index fd13be3..6a0e6fe 100644 --- a/Lib/http/server.py +++ b/Lib/http/server.py @@ -1167,8 +1167,7 @@ def test(HandlerClass=BaseHTTPRequestHandler, ServerClass=HTTPServer, protocol="HTTP/1.0", port=8000, bind=""): """Test the HTTP request handler class. - This runs an HTTP server on port 8000 (or the first command line - argument). + This runs an HTTP server on port 8000 (or the port argument). """ server_address = (bind, port) diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt index 2d8ce54..be063c4 100644 --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -2,6 +2,19 @@ What's New in IDLE 3.5.0? ========================= *Release date: 2015-09-13* ?? +- Issue #23672: Allow Idle to edit and run files with astral chars in name. + Patch by Mohd Sanad Zaki Rizvi. + +- Issue 24745: Idle editor default font. Switch from Courier to + platform-sensitive TkFixedFont. This should not affect current customized + font selections. If there is a problem, edit $HOME/.idlerc/config-main.cfg + and remove 'fontxxx' entries from [Editor Window]. Patch by Mark Roseman. + +- Issue #21192: Idle editor. When a file is run, put its name in the restart bar. + Do not print false prompts. Original patch by Adnan Umer. + +- Issue #13884: Idle menus. Remove tearoff lines. Patch by Roger Serwy. + - Issue #23184: remove unused names and imports in idlelib. Initial patch by Al Sweigart. diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py index 5854cf9..90fc689 100755 --- a/Lib/idlelib/PyShell.py +++ b/Lib/idlelib/PyShell.py @@ -1043,6 +1043,7 @@ class PyShell(OutputWindow): self.write("Python %s on %s\n%s\n%s" % (sys.version, sys.platform, self.COPYRIGHT, nosub)) + self.text.focus_force() self.showprompt() import tkinter tkinter._default_root = None # 03Jan04 KBK What's this? diff --git a/Lib/idlelib/ScriptBinding.py b/Lib/idlelib/ScriptBinding.py index e8cb2fc..e5636df 100644 --- a/Lib/idlelib/ScriptBinding.py +++ b/Lib/idlelib/ScriptBinding.py @@ -69,7 +69,7 @@ class ScriptBinding: try: tabnanny.process_tokens(tokenize.generate_tokens(f.readline)) except tokenize.TokenError as msg: - msgtxt, (lineno, start) = msg + msgtxt, (lineno, start) = msg.args self.editwin.gotoline(lineno) self.errorbox("Tabnanny Tokenizing Error", "Token Error: %s" % msgtxt) diff --git a/Lib/idlelib/StackViewer.py b/Lib/idlelib/StackViewer.py index b1e5e26..ccc755c 100644 --- a/Lib/idlelib/StackViewer.py +++ b/Lib/idlelib/StackViewer.py @@ -10,8 +10,7 @@ from idlelib.PyShell import PyShellFileList def StackBrowser(root, flist=None, tb=None, top=None): if top is None: - from tkinter import Toplevel - top = Toplevel(root) + top = tk.Toplevel(root) sc = ScrolledCanvas(top, bg="white", highlightthickness=0) sc.frame.pack(expand=1, fill="both") item = StackTreeItem(flist, tb) @@ -108,12 +107,9 @@ class VariablesTreeItem(ObjectTreeItem): def IsExpandable(self): return len(self.object) > 0 - def keys(self): - return list(self.object.keys()) - def GetSubList(self): sublist = [] - for key in self.keys(): + for key in self.object.keys(): try: value = self.object[key] except KeyError: @@ -124,6 +120,9 @@ class VariablesTreeItem(ObjectTreeItem): sublist.append(item) return sublist + def keys(self): # unused, left for possible 3rd party use + return list(self.object.keys()) + def _stack_viewer(parent): root = tk.Tk() root.title("Test StackViewer") diff --git a/Lib/idlelib/configDialog.py b/Lib/idlelib/configDialog.py index 9ed6336..b70cb60 100644 --- a/Lib/idlelib/configDialog.py +++ b/Lib/idlelib/configDialog.py @@ -1201,9 +1201,6 @@ class VerticalScrolledFrame(Frame): # update the scrollbars to match the size of the inner frame size = (interior.winfo_reqwidth(), interior.winfo_reqheight()) canvas.config(scrollregion="0 0 %s %s" % size) - if interior.winfo_reqwidth() != canvas.winfo_width(): - # update the canvas's width to fit the inner frame - canvas.config(width=interior.winfo_reqwidth()) interior.bind('<Configure>', _configure_interior) def _configure_canvas(event): @@ -1323,38 +1320,56 @@ class ConfigExtensionsDialog(Toplevel): def create_widgets(self): """Create the dialog's widgets.""" + self.extension_names = StringVar(self) self.rowconfigure(0, weight=1) - self.rowconfigure(1, weight=0) - self.columnconfigure(0, weight=1) - - # create the tabbed pages - self.tabbed_page_set = TabbedPageSet( - self, page_names=self.extensions.keys(), - n_rows=None, max_tabs_per_row=5, - page_class=TabbedPageSet.PageRemove) - self.tabbed_page_set.grid(row=0, column=0, sticky=NSEW) - for ext_name in self.extensions: - self.create_tab_page(ext_name) - - self.create_action_buttons().grid(row=1) + self.columnconfigure(2, weight=1) + self.extension_list = Listbox(self, listvariable=self.extension_names, + selectmode='browse') + self.extension_list.bind('<<ListboxSelect>>', self.extension_selected) + scroll = Scrollbar(self, command=self.extension_list.yview) + self.extension_list.yscrollcommand=scroll.set + self.details_frame = LabelFrame(self, width=250, height=250) + self.extension_list.grid(column=0, row=0, sticky='nws') + scroll.grid(column=1, row=0, sticky='ns') + self.details_frame.grid(column=2, row=0, sticky='nsew', padx=[10, 0]) + self.configure(padx=10, pady=10) + self.config_frame = {} + self.current_extension = None + + self.outerframe = self # TEMPORARY + self.tabbed_page_set = self.extension_list # TEMPORARY + + # create the individual pages + ext_names = '' + for ext_name in sorted(self.extensions): + self.create_extension_frame(ext_name) + ext_names = ext_names + '{' + ext_name + '} ' + self.extension_names.set(ext_names) + self.extension_list.selection_set(0) + self.extension_selected(None) + self.create_action_buttons().grid(row=1, columnspan=3) + + def extension_selected(self, event): + newsel = self.extension_list.curselection() + if newsel: + newsel = self.extension_list.get(newsel) + if newsel is None or newsel != self.current_extension: + if self.current_extension: + self.details_frame.config(text='') + self.config_frame[self.current_extension].grid_forget() + self.current_extension = None + if newsel: + self.details_frame.config(text=newsel) + self.config_frame[newsel].grid(column=0, row=0, sticky='nsew') + self.current_extension = newsel create_action_buttons = ConfigDialog.create_action_buttons - def create_tab_page(self, ext_name): - """Create the page for an extension.""" - - page = LabelFrame(self.tabbed_page_set.pages[ext_name].frame, - border=2, padx=2, relief=GROOVE, - text=' %s ' % ext_name) - page.pack(fill=BOTH, expand=True, padx=12, pady=2) - - # create the scrollable frame which will contain the entries - scrolled_frame = VerticalScrolledFrame(page, pady=2, height=250) - scrolled_frame.pack(side=BOTTOM, fill=BOTH, expand=TRUE) - entry_area = scrolled_frame.interior - entry_area.columnconfigure(0, weight=0) - entry_area.columnconfigure(1, weight=1) - + def create_extension_frame(self, ext_name): + """Create a frame holding the widgets to configure one extension""" + f = VerticalScrolledFrame(self.details_frame, height=250, width=250) + self.config_frame[ext_name] = f + entry_area = f.interior # create an entry for each configuration option for row, opt in enumerate(self.extensions[ext_name]): # create a row with a label and entry/checkbutton @@ -1365,15 +1380,15 @@ class ConfigExtensionsDialog(Toplevel): Checkbutton(entry_area, textvariable=var, variable=var, onvalue='True', offvalue='False', indicatoron=FALSE, selectcolor='', width=8 - ).grid(row=row, column=1, sticky=W, padx=7) + ).grid(row=row, column=1, sticky=W, padx=7) elif opt['type'] == 'int': Entry(entry_area, textvariable=var, validate='key', - validatecommand=(self.is_int, '%P') - ).grid(row=row, column=1, sticky=NSEW, padx=7) + validatecommand=(self.is_int, '%P') + ).grid(row=row, column=1, sticky=NSEW, padx=7) else: Entry(entry_area, textvariable=var - ).grid(row=row, column=1, sticky=NSEW, padx=7) + ).grid(row=row, column=1, sticky=NSEW, padx=7) return diff --git a/Lib/idlelib/idle_test/test_warning.py b/Lib/idlelib/idle_test/test_warning.py index 18627dd..54ac993 100644 --- a/Lib/idlelib/idle_test/test_warning.py +++ b/Lib/idlelib/idle_test/test_warning.py @@ -68,6 +68,15 @@ class ShellWarnTest(unittest.TestCase): 'Test', UserWarning, 'test_warning.py', 99, f, 'Line of code') self.assertEqual(shellmsg.splitlines(), f.getvalue().splitlines()) +class ImportWarnTest(unittest.TestCase): + def test_idlever(self): + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter("always") + import idlelib.idlever + self.assertEqual(len(w), 1) + self.assertTrue(issubclass(w[-1].category, DeprecationWarning)) + self.assertIn("version", str(w[-1].message)) + if __name__ == '__main__': unittest.main(verbosity=2, exit=False) diff --git a/Lib/idlelib/idlever.py b/Lib/idlelib/idlever.py index 563d933..13c68b8 100644 --- a/Lib/idlelib/idlever.py +++ b/Lib/idlelib/idlever.py @@ -1,4 +1,12 @@ -"""Unused by Idle: there is no separate Idle version anymore. -Kept only for possible existing extension use.""" +""" +The separate Idle version was eliminated years ago; +idlelib.idlever is no longer used by Idle +and will be removed in 3.6 or later. Use + from sys import version + IDLE_VERSION = version[:version.index(' ')] +""" +# Kept for now only for possible existing extension use +import warnings as w +w.warn(__doc__, DeprecationWarning) from sys import version IDLE_VERSION = version[:version.index(' ')] @@ -1669,6 +1669,9 @@ def main(): # In most cases SystemExit does not warrant a post-mortem session. print("The program exited via sys.exit(). Exit status:", end=' ') print(sys.exc_info()[1]) + except SyntaxError: + traceback.print_exc() + sys.exit(1) except: traceback.print_exc() print("Uncaught exception. Entering post mortem debugging") diff --git a/Lib/shutil.py b/Lib/shutil.py index a5da587..3f4b6bf 100644 --- a/Lib/shutil.py +++ b/Lib/shutil.py @@ -679,7 +679,16 @@ def _make_zipfile(base_name, base_dir, verbose=0, dry_run=0, logger=None): if not dry_run: with zipfile.ZipFile(zip_filename, "w", compression=zipfile.ZIP_DEFLATED) as zf: + path = os.path.normpath(base_dir) + zf.write(path, path) + if logger is not None: + logger.info("adding '%s'", path) for dirpath, dirnames, filenames in os.walk(base_dir): + for name in sorted(dirnames): + path = os.path.normpath(os.path.join(dirpath, name)) + zf.write(path, path) + if logger is not None: + logger.info("adding '%s'", path) for name in filenames: path = os.path.normpath(os.path.join(dirpath, name)) if os.path.isfile(path): diff --git a/Lib/test/test_binascii.py b/Lib/test/test_binascii.py index 4e67887..8367afe 100644 --- a/Lib/test/test_binascii.py +++ b/Lib/test/test_binascii.py @@ -178,6 +178,8 @@ class BinASCIITest(unittest.TestCase): self.assertEqual(binascii.unhexlify(self.type2test(t)), u) def test_qp(self): + binascii.a2b_qp(data=b"", header=False) # Keyword arguments allowed + # A test for SF bug 534347 (segfaults without the proper fix) try: binascii.a2b_qp(b"", **{1:1}) @@ -185,6 +187,7 @@ class BinASCIITest(unittest.TestCase): pass else: self.fail("binascii.a2b_qp(**{1:1}) didn't raise TypeError") + self.assertEqual(binascii.a2b_qp(b"= "), b"= ") self.assertEqual(binascii.a2b_qp(b"=="), b"=") self.assertEqual(binascii.a2b_qp(b"=AX"), b"=AX") diff --git a/Lib/test/test_cgi.py b/Lib/test/test_cgi.py index a7a9d02..ab9f6ab 100644 --- a/Lib/test/test_cgi.py +++ b/Lib/test/test_cgi.py @@ -326,6 +326,24 @@ Content-Type: text/plain got = getattr(files[x], k) self.assertEqual(got, exp) + def test_fieldstorage_part_content_length(self): + BOUNDARY = "JfISa01" + POSTDATA = """--JfISa01 +Content-Disposition: form-data; name="submit-name" +Content-Length: 5 + +Larry +--JfISa01""" + env = { + 'REQUEST_METHOD': 'POST', + 'CONTENT_TYPE': 'multipart/form-data; boundary={}'.format(BOUNDARY), + 'CONTENT_LENGTH': str(len(POSTDATA))} + fp = BytesIO(POSTDATA.encode('latin-1')) + fs = cgi.FieldStorage(fp, environ=env, encoding="latin-1") + self.assertEqual(len(fs.list), 1) + self.assertEqual(fs.list[0].name, 'submit-name') + self.assertEqual(fs.list[0].value, 'Larry') + def test_fieldstorage_as_context_manager(self): fp = BytesIO(b'x' * 10) env = {'REQUEST_METHOD': 'PUT'} diff --git a/Lib/test/test_collections.py b/Lib/test/test_collections.py index c2d03ee..4124f91 100644 --- a/Lib/test/test_collections.py +++ b/Lib/test/test_collections.py @@ -257,7 +257,6 @@ class TestNamedTuple(unittest.TestCase): self.assertEqual(p._fields, ('x', 'y')) # test _fields attribute self.assertEqual(p._replace(x=1), (1, 22)) # test _replace method self.assertEqual(p._asdict(), dict(x=11, y=22)) # test _asdict method - self.assertEqual(vars(p), p._asdict()) # verify that vars() works try: p._replace(x=1, error=2) @@ -412,6 +411,17 @@ class TestNamedTuple(unittest.TestCase): globals().pop('NTColor', None) # clean-up after this test + def test_namedtuple_subclass_issue_24931(self): + class Point(namedtuple('_Point', ['x', 'y'])): + pass + + a = Point(3, 4) + self.assertEqual(a._asdict(), OrderedDict([('x', 3), ('y', 4)])) + + a.w = 5 + self.assertEqual(a.__dict__, {'w': 5}) + + ################################################################################ ### Abstract Base Classes ################################################################################ diff --git a/Lib/test/test_configparser.py b/Lib/test/test_configparser.py index 470d2cd..71a8f3f 100644 --- a/Lib/test/test_configparser.py +++ b/Lib/test/test_configparser.py @@ -847,7 +847,8 @@ class ConfigParserTestCase(BasicTestCase, unittest.TestCase): "something with lots of interpolation (10 steps)") e = self.get_error(cf, configparser.InterpolationDepthError, "Foo", "bar11") if self.interpolation == configparser._UNSET: - self.assertEqual(e.args, ("bar11", "Foo", "%(with1)s")) + self.assertEqual(e.args, ("bar11", "Foo", + "something %(with11)s lots of interpolation (11 steps)")) elif isinstance(self.interpolation, configparser.LegacyInterpolation): self.assertEqual(e.args, ("bar11", "Foo", "something %(with11)s lots of interpolation (11 steps)")) @@ -861,7 +862,7 @@ class ConfigParserTestCase(BasicTestCase, unittest.TestCase): self.assertEqual(e.option, "name") if self.interpolation == configparser._UNSET: self.assertEqual(e.args, ('name', 'Interpolation Error', - '', 'reference')) + '%(reference)s', 'reference')) elif isinstance(self.interpolation, configparser.LegacyInterpolation): self.assertEqual(e.args, ('name', 'Interpolation Error', '%(reference)s', 'reference')) @@ -1177,7 +1178,7 @@ class ConfigParserTestCaseExtendedInterpolation(BasicTestCase, unittest.TestCase with self.assertRaises(exception_class) as cm: cf['interpolated']['$trying'] self.assertEqual(cm.exception.reference, 'dollars:${sick') - self.assertEqual(cm.exception.args[2], '}') #rawval + self.assertEqual(cm.exception.args[2], '${dollars:${sick}}') #rawval def test_case_sensitivity_basic(self): ini = textwrap.dedent(""" diff --git a/Lib/test/test_functools.py b/Lib/test/test_functools.py index ac211c4..ae929ec 100644 --- a/Lib/test/test_functools.py +++ b/Lib/test/test_functools.py @@ -1491,6 +1491,24 @@ class TestSingleDispatch(unittest.TestCase): many_abcs = [c.Mapping, c.Sized, c.Callable, c.Container, c.Iterable] self.assertEqual(mro(X, abcs=many_abcs), expected) + def test_false_meta(self): + # see issue23572 + class MetaA(type): + def __len__(self): + return 0 + class A(metaclass=MetaA): + pass + class AA(A): + pass + @functools.singledispatch + def fun(a): + return 'base A' + @fun.register(A) + def _(a): + return 'fun A' + aa = AA() + self.assertEqual(fun(aa), 'fun A') + def test_mro_conflicts(self): c = collections @functools.singledispatch diff --git a/Lib/test/test_gdb.py b/Lib/test/test_gdb.py index 0322677..db777be 100644 --- a/Lib/test/test_gdb.py +++ b/Lib/test/test_gdb.py @@ -21,19 +21,34 @@ except ImportError: from test import support from test.support import run_unittest, findfile, python_is_optimized -try: - gdb_version, _ = subprocess.Popen(["gdb", "-nx", "--version"], - stdout=subprocess.PIPE).communicate() -except OSError: - # This is what "no gdb" looks like. There may, however, be other - # errors that manifest this way too. - raise unittest.SkipTest("Couldn't find gdb on the path") -gdb_version_number = re.search(b"^GNU gdb [^\d]*(\d+)\.(\d)", gdb_version) -gdb_major_version = int(gdb_version_number.group(1)) -gdb_minor_version = int(gdb_version_number.group(2)) +def get_gdb_version(): + try: + proc = subprocess.Popen(["gdb", "-nx", "--version"], + stdout=subprocess.PIPE, + universal_newlines=True) + with proc: + version = proc.communicate()[0] + except OSError: + # This is what "no gdb" looks like. There may, however, be other + # errors that manifest this way too. + raise unittest.SkipTest("Couldn't find gdb on the path") + + # Regex to parse: + # 'GNU gdb (GDB; SUSE Linux Enterprise 12) 7.7\n' -> 7.7 + # 'GNU gdb (GDB) Fedora 7.9.1-17.fc22\n' -> 7.9 + # 'GNU gdb 6.1.1 [FreeBSD]\n' -> 6.1 + # 'GNU gdb (GDB) Fedora (7.5.1-37.fc18)\n' -> 7.5 + match = re.search(r"^GNU gdb.*?\b(\d+)\.(\d)", version) + if match is None: + raise Exception("unable to parse GDB version: %r" % version) + return (version, int(match.group(1)), int(match.group(2))) + +gdb_version, gdb_major_version, gdb_minor_version = get_gdb_version() if gdb_major_version < 7: - raise unittest.SkipTest("gdb versions before 7.0 didn't support python embedding" - " Saw:\n" + gdb_version.decode('ascii', 'replace')) + raise unittest.SkipTest("gdb versions before 7.0 didn't support python " + "embedding. Saw %s.%s:\n%s" + % (gdb_major_version, gdb_minor_version, + gdb_version)) if not sysconfig.is_python_build(): raise unittest.SkipTest("test_gdb only works on source builds at the moment.") @@ -59,9 +74,12 @@ def run_gdb(*args, **env_vars): base_cmd = ('gdb', '--batch', '-nx') if (gdb_major_version, gdb_minor_version) >= (7, 4): base_cmd += ('-iex', 'add-auto-load-safe-path ' + checkout_hook_path) - out, err = subprocess.Popen(base_cmd + args, - stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env, - ).communicate() + proc = subprocess.Popen(base_cmd + args, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + env=env) + with proc: + out, err = proc.communicate() return out.decode('utf-8', 'replace'), err.decode('utf-8', 'replace') # Verify that "gdb" was built with the embedded python support enabled: @@ -880,8 +898,8 @@ class PyLocalsTests(DebuggerTests): def test_main(): if support.verbose: - print("GDB version:") - for line in os.fsdecode(gdb_version).splitlines(): + print("GDB version %s.%s:" % (gdb_major_version, gdb_minor_version)) + for line in gdb_version.splitlines(): print(" " * 4 + line) run_unittest(PrettyPrintTests, PyListTests, diff --git a/Lib/test/test_glob.py b/Lib/test/test_glob.py index 21b0153..926588e 100644 --- a/Lib/test/test_glob.py +++ b/Lib/test/test_glob.py @@ -242,9 +242,7 @@ class GlobTests(unittest.TestCase): ('a', 'bcd', 'EF'), ('a', 'bcd', 'efg'))) eq(self.rglob('a', '**', 'bcd'), self.joins(('a', 'bcd'))) - predir = os.path.abspath(os.curdir) - try: - os.chdir(self.tempdir) + with change_cwd(self.tempdir): join = os.path.join eq(glob.glob('**', recursive=True), [join(*i) for i in full]) eq(glob.glob(join('**', ''), recursive=True), @@ -256,8 +254,6 @@ class GlobTests(unittest.TestCase): if can_symlink(): expect += [join('sym3', 'EF')] eq(glob.glob(join('**', 'EF'), recursive=True), expect) - finally: - os.chdir(predir) @skip_unless_symlink diff --git a/Lib/test/test_htmlparser.py b/Lib/test/test_htmlparser.py index de8f3e8..11420b2c 100644 --- a/Lib/test/test_htmlparser.py +++ b/Lib/test/test_htmlparser.py @@ -72,9 +72,6 @@ class EventCollectorExtra(EventCollector): class EventCollectorCharrefs(EventCollector): - def get_events(self): - return self.events - def handle_charref(self, data): self.fail('This should never be called with convert_charrefs=True') @@ -633,6 +630,18 @@ text ] self._run_check(html, expected) + def test_convert_charrefs_dropped_text(self): + # #23144: make sure that all the events are triggered when + # convert_charrefs is True, even if we don't call .close() + parser = EventCollector(convert_charrefs=True) + # before the fix, bar & baz was missing + parser.feed("foo <a>link</a> bar & baz") + self.assertEqual( + parser.get_events(), + [('data', 'foo '), ('starttag', 'a', []), ('data', 'link'), + ('endtag', 'a'), ('data', ' bar & baz')] + ) + class AttributesTestCase(TestCaseBase): diff --git a/Lib/test/test_mmap.py b/Lib/test/test_mmap.py index 3de84e8..0f25742 100644 --- a/Lib/test/test_mmap.py +++ b/Lib/test/test_mmap.py @@ -731,7 +731,10 @@ class LargeMmapTests(unittest.TestCase): f.write(tail) f.flush() except (OSError, OverflowError): - f.close() + try: + f.close() + except (OSError, OverflowError): + pass raise unittest.SkipTest("filesystem does not have largefile support") return f diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py index d91f58c..0b77835 100644 --- a/Lib/test/test_os.py +++ b/Lib/test/test_os.py @@ -85,7 +85,7 @@ HAVE_WHEEL_GROUP = sys.platform.startswith('freebsd') and os.getgid() == 0 # Tests creating TESTFN class FileTests(unittest.TestCase): def setUp(self): - if os.path.exists(support.TESTFN): + if os.path.lexists(support.TESTFN): os.unlink(support.TESTFN) tearDown = setUp @@ -209,6 +209,19 @@ class FileTests(unittest.TestCase): with open(TESTFN2, 'r') as f: self.assertEqual(f.read(), "1") + def test_open_keywords(self): + f = os.open(path=__file__, flags=os.O_RDONLY, mode=0o777, + dir_fd=None) + os.close(f) + + def test_symlink_keywords(self): + symlink = support.get_attribute(os, "symlink") + try: + symlink(src='target', dst=support.TESTFN, + target_is_directory=False, dir_fd=None) + except (NotImplementedError, OSError): + pass # No OS support or unprivileged user + # Test attributes on return values from os.*stat* family. class StatAttributeTests(unittest.TestCase): @@ -2313,6 +2326,14 @@ class TestSendfile(unittest.TestCase): os.sendfile(self.sockno, self.fileno, -1, 4096) self.assertEqual(cm.exception.errno, errno.EINVAL) + def test_keywords(self): + # Keyword arguments should be supported + os.sendfile(out=self.sockno, offset=0, count=4096, + **{'in': self.fileno}) + if self.SUPPORT_HEADERS_TRAILERS: + os.sendfile(self.sockno, self.fileno, offset=0, count=4096, + headers=(), trailers=(), flags=0) + # --- headers / trailers tests @requires_headers_trailers diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py index ec8346c..35044ad 100644 --- a/Lib/test/test_pdb.py +++ b/Lib/test/test_pdb.py @@ -1043,6 +1043,18 @@ class PdbTestCase(unittest.TestCase): self.assertNotIn('Error', stdout.decode(), "Got an error running test script under PDB") + def test_issue16180(self): + # A syntax error in the debuggee. + script = "def f: pass\n" + commands = '' + expected = "SyntaxError:" + stdout, stderr = self.run_pdb(script, commands) + self.assertIn(expected, stdout, + '\n\nExpected:\n{}\nGot:\n{}\n' + 'Fail to handle a syntax error in the debuggee.' + .format(expected, stdout)) + + def tearDown(self): support.unlink(support.TESTFN) diff --git a/Lib/test/test_pep277.py b/Lib/test/test_pep277.py index 6c833a8..98c716b 100644 --- a/Lib/test/test_pep277.py +++ b/Lib/test/test_pep277.py @@ -158,17 +158,11 @@ class UnicodeFileTests(unittest.TestCase): def test_directory(self): dirname = os.path.join(support.TESTFN, 'Gr\xfc\xdf-\u66e8\u66e9\u66eb') filename = '\xdf-\u66e8\u66e9\u66eb' - oldwd = os.getcwd() - os.mkdir(dirname) - os.chdir(dirname) - try: + with support.temp_cwd(dirname): with open(filename, 'wb') as f: f.write((filename + '\n').encode("utf-8")) os.access(filename,os.R_OK) os.remove(filename) - finally: - os.chdir(oldwd) - os.rmdir(dirname) class UnicodeNFCFileTests(UnicodeFileTests): diff --git a/Lib/test/test_popen.py b/Lib/test/test_popen.py index 8958db0..da01a87 100644 --- a/Lib/test/test_popen.py +++ b/Lib/test/test_popen.py @@ -57,5 +57,9 @@ class PopenTest(unittest.TestCase): with os.popen("echo hello") as f: self.assertEqual(list(f), ["hello\n"]) + def test_keywords(self): + with os.popen(cmd="exit 0", mode="w", buffering=-1): + pass + if __name__ == "__main__": unittest.main() diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py index 77e5b0c4..2a59c38 100644 --- a/Lib/test/test_posix.py +++ b/Lib/test/test_posix.py @@ -442,6 +442,14 @@ class PosixTester(unittest.TestCase): else: self.assertTrue(stat.S_ISFIFO(posix.stat(support.TESTFN).st_mode)) + # Keyword arguments are also supported + support.unlink(support.TESTFN) + try: + posix.mknod(path=support.TESTFN, mode=mode, device=0, + dir_fd=None) + except OSError as e: + self.assertIn(e.errno, (errno.EPERM, errno.EINVAL)) + @unittest.skipUnless(hasattr(posix, 'stat'), 'test needs posix.stat()') @unittest.skipUnless(hasattr(posix, 'makedev'), 'test needs posix.makedev()') def test_makedev(self): diff --git a/Lib/test/test_posixpath.py b/Lib/test/test_posixpath.py index ece3555..9d20471 100644 --- a/Lib/test/test_posixpath.py +++ b/Lib/test/test_posixpath.py @@ -316,7 +316,6 @@ class PosixPathTest(unittest.TestCase): # Bug #930024, return the path unchanged if we get into an infinite # symlink loop. try: - old_path = abspath('.') os.symlink(ABSTFN, ABSTFN) self.assertEqual(realpath(ABSTFN), ABSTFN) @@ -342,10 +341,9 @@ class PosixPathTest(unittest.TestCase): self.assertEqual(realpath(ABSTFN+"c"), ABSTFN+"c") # Test using relative path as well. - os.chdir(dirname(ABSTFN)) - self.assertEqual(realpath(basename(ABSTFN)), ABSTFN) + with support.change_cwd(dirname(ABSTFN)): + self.assertEqual(realpath(basename(ABSTFN)), ABSTFN) finally: - os.chdir(old_path) support.unlink(ABSTFN) support.unlink(ABSTFN+"1") support.unlink(ABSTFN+"2") @@ -373,7 +371,6 @@ class PosixPathTest(unittest.TestCase): @skip_if_ABSTFN_contains_backslash def test_realpath_deep_recursion(self): depth = 10 - old_path = abspath('.') try: os.mkdir(ABSTFN) for i in range(depth): @@ -382,10 +379,9 @@ class PosixPathTest(unittest.TestCase): self.assertEqual(realpath(ABSTFN + '/%d' % depth), ABSTFN) # Test using relative path as well. - os.chdir(ABSTFN) - self.assertEqual(realpath('%d' % depth), ABSTFN) + with support.change_cwd(ABSTFN): + self.assertEqual(realpath('%d' % depth), ABSTFN) finally: - os.chdir(old_path) for i in range(depth + 1): support.unlink(ABSTFN + '/%d' % i) safe_rmdir(ABSTFN) @@ -399,15 +395,13 @@ class PosixPathTest(unittest.TestCase): # /usr/doc with 'doc' being a symlink to /usr/share/doc. We call # realpath("a"). This should return /usr/share/doc/a/. try: - old_path = abspath('.') os.mkdir(ABSTFN) os.mkdir(ABSTFN + "/y") os.symlink(ABSTFN + "/y", ABSTFN + "/k") - os.chdir(ABSTFN + "/k") - self.assertEqual(realpath("a"), ABSTFN + "/y/a") + with support.change_cwd(ABSTFN + "/k"): + self.assertEqual(realpath("a"), ABSTFN + "/y/a") finally: - os.chdir(old_path) support.unlink(ABSTFN + "/k") safe_rmdir(ABSTFN + "/y") safe_rmdir(ABSTFN) @@ -424,7 +418,6 @@ class PosixPathTest(unittest.TestCase): # and a symbolic link 'link-y' pointing to 'y' in directory 'a', # then realpath("link-y/..") should return 'k', not 'a'. try: - old_path = abspath('.') os.mkdir(ABSTFN) os.mkdir(ABSTFN + "/k") os.mkdir(ABSTFN + "/k/y") @@ -433,11 +426,10 @@ class PosixPathTest(unittest.TestCase): # Absolute path. self.assertEqual(realpath(ABSTFN + "/link-y/.."), ABSTFN + "/k") # Relative path. - os.chdir(dirname(ABSTFN)) - self.assertEqual(realpath(basename(ABSTFN) + "/link-y/.."), - ABSTFN + "/k") + with support.change_cwd(dirname(ABSTFN)): + self.assertEqual(realpath(basename(ABSTFN) + "/link-y/.."), + ABSTFN + "/k") finally: - os.chdir(old_path) support.unlink(ABSTFN + "/link-y") safe_rmdir(ABSTFN + "/k/y") safe_rmdir(ABSTFN + "/k") @@ -451,17 +443,14 @@ class PosixPathTest(unittest.TestCase): # must be resolved too. try: - old_path = abspath('.') os.mkdir(ABSTFN) os.mkdir(ABSTFN + "/k") os.symlink(ABSTFN, ABSTFN + "link") - os.chdir(dirname(ABSTFN)) - - base = basename(ABSTFN) - self.assertEqual(realpath(base + "link"), ABSTFN) - self.assertEqual(realpath(base + "link/k"), ABSTFN + "/k") + with support.change_cwd(dirname(ABSTFN)): + base = basename(ABSTFN) + self.assertEqual(realpath(base + "link"), ABSTFN) + self.assertEqual(realpath(base + "link/k"), ABSTFN + "/k") finally: - os.chdir(old_path) support.unlink(ABSTFN + "link") safe_rmdir(ABSTFN + "/k") safe_rmdir(ABSTFN) diff --git a/Lib/test/test_py_compile.py b/Lib/test/test_py_compile.py index 03cca5d..4a6caa5 100644 --- a/Lib/test/test_py_compile.py +++ b/Lib/test/test_py_compile.py @@ -63,11 +63,9 @@ class PyCompileTests(unittest.TestCase): self.assertTrue(os.path.exists(self.cache_path)) def test_cwd(self): - cwd = os.getcwd() - os.chdir(self.directory) - py_compile.compile(os.path.basename(self.source_path), - os.path.basename(self.pyc_path)) - os.chdir(cwd) + with support.change_cwd(self.directory): + py_compile.compile(os.path.basename(self.source_path), + os.path.basename(self.pyc_path)) self.assertTrue(os.path.exists(self.pyc_path)) self.assertFalse(os.path.exists(self.cache_path)) diff --git a/Lib/test/test_pyexpat.py b/Lib/test/test_pyexpat.py index 08e95c6..550aebf 100644 --- a/Lib/test/test_pyexpat.py +++ b/Lib/test/test_pyexpat.py @@ -3,6 +3,7 @@ from io import BytesIO import os +import sys import sysconfig import unittest import traceback @@ -16,22 +17,47 @@ from test.support import sortdict class SetAttributeTest(unittest.TestCase): def setUp(self): self.parser = expat.ParserCreate(namespace_separator='!') - self.set_get_pairs = [ - [0, 0], - [1, 1], - [2, 1], - [0, 0], - ] + + def test_buffer_text(self): + self.assertIs(self.parser.buffer_text, False) + for x in 0, 1, 2, 0: + self.parser.buffer_text = x + self.assertIs(self.parser.buffer_text, bool(x)) + + def test_namespace_prefixes(self): + self.assertIs(self.parser.namespace_prefixes, False) + for x in 0, 1, 2, 0: + self.parser.namespace_prefixes = x + self.assertIs(self.parser.namespace_prefixes, bool(x)) def test_ordered_attributes(self): - for x, y in self.set_get_pairs: + self.assertIs(self.parser.ordered_attributes, False) + for x in 0, 1, 2, 0: self.parser.ordered_attributes = x - self.assertEqual(self.parser.ordered_attributes, y) + self.assertIs(self.parser.ordered_attributes, bool(x)) + + def test_specified_attributes(self): + self.assertIs(self.parser.specified_attributes, False) + for x in 0, 1, 2, 0: + self.parser.specified_attributes = x + self.assertIs(self.parser.specified_attributes, bool(x)) def test_specified_attributes(self): - for x, y in self.set_get_pairs: + self.assertIs(self.parser.specified_attributes, False) + for x in 0, 1, 2, 0: self.parser.specified_attributes = x - self.assertEqual(self.parser.specified_attributes, y) + self.assertIs(self.parser.specified_attributes, bool(x)) + + def test_invalid_attributes(self): + with self.assertRaises(AttributeError): + self.parser.returns_unicode = 1 + with self.assertRaises(AttributeError): + self.parser.returns_unicode + + # Issue #25019 + self.assertRaises(TypeError, setattr, self.parser, range(0xF), 0) + self.assertRaises(TypeError, self.parser.__setattr__, range(0xF), 0) + self.assertRaises(TypeError, getattr, self.parser, range(0xF)) data = b'''\ @@ -514,11 +540,14 @@ class ChardataBufferTest(unittest.TestCase): def test_wrong_size(self): parser = expat.ParserCreate() parser.buffer_text = 1 - def f(size): - parser.buffer_size = size - - self.assertRaises(ValueError, f, -1) - self.assertRaises(ValueError, f, 0) + with self.assertRaises(ValueError): + parser.buffer_size = -1 + with self.assertRaises(ValueError): + parser.buffer_size = 0 + with self.assertRaises((ValueError, OverflowError)): + parser.buffer_size = sys.maxsize + 1 + with self.assertRaises(TypeError): + parser.buffer_size = 512.0 def test_unchanged_size(self): xml1 = b"<?xml version='1.0' encoding='iso8859'?><s>" + b'a' * 512 diff --git a/Lib/test/test_shutil.py b/Lib/test/test_shutil.py index 5183c3c..522959a 100644 --- a/Lib/test/test_shutil.py +++ b/Lib/test/test_shutil.py @@ -12,11 +12,9 @@ import errno import functools import subprocess from contextlib import ExitStack -from test import support -from test.support import TESTFN from os.path import splitdrive from distutils.spawn import find_executable, spawn -from shutil import (_make_tarball, _make_zipfile, make_archive, +from shutil import (make_archive, register_archive_format, unregister_archive_format, get_archive_formats, Error, unpack_archive, register_unpack_format, RegistryError, @@ -94,6 +92,18 @@ def read_file(path, binary=False): with open(path, 'rb' if binary else 'r') as fp: return fp.read() +def rlistdir(path): + res = [] + for name in sorted(os.listdir(path)): + p = os.path.join(path, name) + if os.path.isdir(p) and not os.path.islink(p): + res.append(name + '/') + for n in rlistdir(p): + res.append(name + '/' + n) + else: + res.append(name) + return res + class TestShutil(unittest.TestCase): @@ -959,139 +969,143 @@ class TestShutil(unittest.TestCase): @requires_zlib def test_make_tarball(self): # creating something to tar - tmpdir = self.mkdtemp() - write_file((tmpdir, 'file1'), 'xxx') - write_file((tmpdir, 'file2'), 'xxx') - os.mkdir(os.path.join(tmpdir, 'sub')) - write_file((tmpdir, 'sub', 'file3'), 'xxx') + root_dir, base_dir = self._create_files('') tmpdir2 = self.mkdtemp() # force shutil to create the directory os.rmdir(tmpdir2) - unittest.skipUnless(splitdrive(tmpdir)[0] == splitdrive(tmpdir2)[0], - "source and target should be on same drive") + # working with relative paths + work_dir = os.path.dirname(tmpdir2) + rel_base_name = os.path.join(os.path.basename(tmpdir2), 'archive') - base_name = os.path.join(tmpdir2, 'archive') - - # working with relative paths to avoid tar warnings - old_dir = os.getcwd() - os.chdir(tmpdir) - try: - _make_tarball(splitdrive(base_name)[1], '.') - finally: - os.chdir(old_dir) + with support.change_cwd(work_dir): + base_name = os.path.abspath(rel_base_name) + tarball = make_archive(rel_base_name, 'gztar', root_dir, '.') # check if the compressed tarball was created - tarball = base_name + '.tar.gz' - self.assertTrue(os.path.exists(tarball)) + self.assertEqual(tarball, base_name + '.tar.gz') + self.assertTrue(os.path.isfile(tarball)) + self.assertTrue(tarfile.is_tarfile(tarball)) + with tarfile.open(tarball, 'r:gz') as tf: + self.assertCountEqual(tf.getnames(), + ['.', './sub', './sub2', + './file1', './file2', './sub/file3']) # trying an uncompressed one - base_name = os.path.join(tmpdir2, 'archive') - old_dir = os.getcwd() - os.chdir(tmpdir) - try: - _make_tarball(splitdrive(base_name)[1], '.', compress=None) - finally: - os.chdir(old_dir) - tarball = base_name + '.tar' - self.assertTrue(os.path.exists(tarball)) + with support.change_cwd(work_dir): + tarball = make_archive(rel_base_name, 'tar', root_dir, '.') + self.assertEqual(tarball, base_name + '.tar') + self.assertTrue(os.path.isfile(tarball)) + self.assertTrue(tarfile.is_tarfile(tarball)) + with tarfile.open(tarball, 'r') as tf: + self.assertCountEqual(tf.getnames(), + ['.', './sub', './sub2', + './file1', './file2', './sub/file3']) def _tarinfo(self, path): - tar = tarfile.open(path) - try: + with tarfile.open(path) as tar: names = tar.getnames() names.sort() return tuple(names) - finally: - tar.close() - def _create_files(self): + def _create_files(self, base_dir='dist'): # creating something to tar - tmpdir = self.mkdtemp() - dist = os.path.join(tmpdir, 'dist') - os.mkdir(dist) + root_dir = self.mkdtemp() + dist = os.path.join(root_dir, base_dir) + os.makedirs(dist, exist_ok=True) write_file((dist, 'file1'), 'xxx') write_file((dist, 'file2'), 'xxx') os.mkdir(os.path.join(dist, 'sub')) write_file((dist, 'sub', 'file3'), 'xxx') os.mkdir(os.path.join(dist, 'sub2')) - tmpdir2 = self.mkdtemp() - base_name = os.path.join(tmpdir2, 'archive') - return tmpdir, tmpdir2, base_name + if base_dir: + write_file((root_dir, 'outer'), 'xxx') + return root_dir, base_dir @requires_zlib - @unittest.skipUnless(find_executable('tar') and find_executable('gzip'), + @unittest.skipUnless(find_executable('tar'), 'Need the tar command to run') def test_tarfile_vs_tar(self): - tmpdir, tmpdir2, base_name = self._create_files() - old_dir = os.getcwd() - os.chdir(tmpdir) - try: - _make_tarball(base_name, 'dist') - finally: - os.chdir(old_dir) + root_dir, base_dir = self._create_files() + base_name = os.path.join(self.mkdtemp(), 'archive') + tarball = make_archive(base_name, 'gztar', root_dir, base_dir) # check if the compressed tarball was created - tarball = base_name + '.tar.gz' - self.assertTrue(os.path.exists(tarball)) + self.assertEqual(tarball, base_name + '.tar.gz') + self.assertTrue(os.path.isfile(tarball)) # now create another tarball using `tar` - tarball2 = os.path.join(tmpdir, 'archive2.tar.gz') - tar_cmd = ['tar', '-cf', 'archive2.tar', 'dist'] - gzip_cmd = ['gzip', '-f9', 'archive2.tar'] - old_dir = os.getcwd() - os.chdir(tmpdir) - try: - with captured_stdout() as s: - spawn(tar_cmd) - spawn(gzip_cmd) - finally: - os.chdir(old_dir) + tarball2 = os.path.join(root_dir, 'archive2.tar') + tar_cmd = ['tar', '-cf', 'archive2.tar', base_dir] + with support.change_cwd(root_dir), captured_stdout(): + spawn(tar_cmd) - self.assertTrue(os.path.exists(tarball2)) + self.assertTrue(os.path.isfile(tarball2)) # let's compare both tarballs self.assertEqual(self._tarinfo(tarball), self._tarinfo(tarball2)) # trying an uncompressed one - base_name = os.path.join(tmpdir2, 'archive') - old_dir = os.getcwd() - os.chdir(tmpdir) - try: - _make_tarball(base_name, 'dist', compress=None) - finally: - os.chdir(old_dir) - tarball = base_name + '.tar' - self.assertTrue(os.path.exists(tarball)) + tarball = make_archive(base_name, 'tar', root_dir, base_dir) + self.assertEqual(tarball, base_name + '.tar') + self.assertTrue(os.path.isfile(tarball)) # now for a dry_run - base_name = os.path.join(tmpdir2, 'archive') - old_dir = os.getcwd() - os.chdir(tmpdir) - try: - _make_tarball(base_name, 'dist', compress=None, dry_run=True) - finally: - os.chdir(old_dir) - tarball = base_name + '.tar' - self.assertTrue(os.path.exists(tarball)) + tarball = make_archive(base_name, 'tar', root_dir, base_dir, + dry_run=True) + self.assertEqual(tarball, base_name + '.tar') + self.assertTrue(os.path.isfile(tarball)) @requires_zlib @unittest.skipUnless(ZIP_SUPPORT, 'Need zip support to run') def test_make_zipfile(self): - # creating something to tar - tmpdir = self.mkdtemp() - write_file((tmpdir, 'file1'), 'xxx') - write_file((tmpdir, 'file2'), 'xxx') + # creating something to zip + root_dir, base_dir = self._create_files() tmpdir2 = self.mkdtemp() # force shutil to create the directory os.rmdir(tmpdir2) - base_name = os.path.join(tmpdir2, 'archive') - _make_zipfile(base_name, tmpdir) + # working with relative paths + work_dir = os.path.dirname(tmpdir2) + rel_base_name = os.path.join(os.path.basename(tmpdir2), 'archive') + + with support.change_cwd(work_dir): + base_name = os.path.abspath(rel_base_name) + res = make_archive(rel_base_name, 'zip', root_dir, base_dir) + + self.assertEqual(res, base_name + '.zip') + self.assertTrue(os.path.isfile(res)) + self.assertTrue(zipfile.is_zipfile(res)) + with zipfile.ZipFile(res) as zf: + self.assertCountEqual(zf.namelist(), + ['dist/', 'dist/sub/', 'dist/sub2/', + 'dist/file1', 'dist/file2', 'dist/sub/file3']) - # check if the compressed tarball was created - tarball = base_name + '.zip' - self.assertTrue(os.path.exists(tarball)) + @requires_zlib + @unittest.skipUnless(ZIP_SUPPORT, 'Need zip support to run') + @unittest.skipUnless(find_executable('zip'), + 'Need the zip command to run') + def test_zipfile_vs_zip(self): + root_dir, base_dir = self._create_files() + base_name = os.path.join(self.mkdtemp(), 'archive') + archive = make_archive(base_name, 'zip', root_dir, base_dir) + + # check if ZIP file was created + self.assertEqual(archive, base_name + '.zip') + self.assertTrue(os.path.isfile(archive)) + + # now create another ZIP file using `zip` + archive2 = os.path.join(root_dir, 'archive2.zip') + zip_cmd = ['zip', '-q', '-r', 'archive2.zip', base_dir] + with support.change_cwd(root_dir): + spawn(zip_cmd) + self.assertTrue(os.path.isfile(archive2)) + # let's compare both ZIP files + with zipfile.ZipFile(archive) as zf: + names = zf.namelist() + with zipfile.ZipFile(archive2) as zf: + names2 = zf.namelist() + self.assertEqual(sorted(names), sorted(names2)) def test_make_archive(self): tmpdir = self.mkdtemp() @@ -1108,40 +1122,37 @@ class TestShutil(unittest.TestCase): else: group = owner = 'root' - base_dir, root_dir, base_name = self._create_files() - base_name = os.path.join(self.mkdtemp() , 'archive') + root_dir, base_dir = self._create_files() + base_name = os.path.join(self.mkdtemp(), 'archive') res = make_archive(base_name, 'zip', root_dir, base_dir, owner=owner, group=group) - self.assertTrue(os.path.exists(res)) + self.assertTrue(os.path.isfile(res)) res = make_archive(base_name, 'zip', root_dir, base_dir) - self.assertTrue(os.path.exists(res)) + self.assertTrue(os.path.isfile(res)) res = make_archive(base_name, 'tar', root_dir, base_dir, owner=owner, group=group) - self.assertTrue(os.path.exists(res)) + self.assertTrue(os.path.isfile(res)) res = make_archive(base_name, 'tar', root_dir, base_dir, owner='kjhkjhkjg', group='oihohoh') - self.assertTrue(os.path.exists(res)) + self.assertTrue(os.path.isfile(res)) @requires_zlib @unittest.skipUnless(UID_GID_SUPPORT, "Requires grp and pwd support") def test_tarfile_root_owner(self): - tmpdir, tmpdir2, base_name = self._create_files() - old_dir = os.getcwd() - os.chdir(tmpdir) + root_dir, base_dir = self._create_files() + base_name = os.path.join(self.mkdtemp(), 'archive') group = grp.getgrgid(0)[0] owner = pwd.getpwuid(0)[0] - try: - archive_name = _make_tarball(base_name, 'dist', compress=None, - owner=owner, group=group) - finally: - os.chdir(old_dir) + with support.change_cwd(root_dir): + archive_name = make_archive(base_name, 'gztar', root_dir, 'dist', + owner=owner, group=group) # check if the compressed tarball was created - self.assertTrue(os.path.exists(archive_name)) + self.assertTrue(os.path.isfile(archive_name)) # now checks the rights archive = tarfile.open(archive_name) @@ -1198,18 +1209,6 @@ class TestShutil(unittest.TestCase): formats = [name for name, params in get_archive_formats()] self.assertNotIn('xxx', formats) - def _compare_dirs(self, dir1, dir2): - # check that dir1 and dir2 are equivalent, - # return the diff - diff = [] - for root, dirs, files in os.walk(dir1): - for file_ in files: - path = os.path.join(root, file_) - target_path = os.path.join(dir2, os.path.split(path)[-1]) - if not os.path.exists(target_path): - diff.append(file_) - return diff - @requires_zlib def test_unpack_archive(self): formats = ['tar', 'gztar', 'zip'] @@ -1218,22 +1217,22 @@ class TestShutil(unittest.TestCase): if LZMA_SUPPORTED: formats.append('xztar') + root_dir, base_dir = self._create_files() + expected = rlistdir(root_dir) + expected.remove('outer') for format in formats: - tmpdir = self.mkdtemp() - base_dir, root_dir, base_name = self._create_files() - tmpdir2 = self.mkdtemp() + base_name = os.path.join(self.mkdtemp(), 'archive') filename = make_archive(base_name, format, root_dir, base_dir) # let's try to unpack it now + tmpdir2 = self.mkdtemp() unpack_archive(filename, tmpdir2) - diff = self._compare_dirs(tmpdir, tmpdir2) - self.assertEqual(diff, []) + self.assertEqual(rlistdir(tmpdir2), expected) # and again, this time with the format specified tmpdir3 = self.mkdtemp() unpack_archive(filename, tmpdir3, format=format) - diff = self._compare_dirs(tmpdir, tmpdir3) - self.assertEqual(diff, []) + self.assertEqual(rlistdir(tmpdir3), expected) self.assertRaises(shutil.ReadError, unpack_archive, TESTFN) self.assertRaises(ValueError, unpack_archive, TESTFN, format='xxx') diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py index 9c0229a..ff2010d 100644 --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -317,11 +317,8 @@ class ProcessTestCase(BaseTestCase): # Normalize an expected cwd (for Tru64 support). # We can't use os.path.realpath since it doesn't expand Tru64 {memb} # strings. See bug #1063571. - original_cwd = os.getcwd() - os.chdir(cwd) - cwd = os.getcwd() - os.chdir(original_cwd) - return cwd + with support.change_cwd(cwd): + return os.getcwd() # For use in the test_cwd* tests below. def _split_python_path(self): diff --git a/Lib/test/test_sysconfig.py b/Lib/test/test_sysconfig.py index c0f27a6..0917c3e 100644 --- a/Lib/test/test_sysconfig.py +++ b/Lib/test/test_sysconfig.py @@ -6,7 +6,7 @@ import shutil from copy import copy from test.support import (run_unittest, TESTFN, unlink, check_warnings, - captured_stdout, skip_unless_symlink) + captured_stdout, skip_unless_symlink, change_cwd) import sysconfig from sysconfig import (get_paths, get_platform, get_config_vars, @@ -361,12 +361,8 @@ class TestSysConfig(unittest.TestCase): # srcdir should be independent of the current working directory # See Issues #15322, #15364. srcdir = sysconfig.get_config_var('srcdir') - cwd = os.getcwd() - try: - os.chdir('..') + with change_cwd(os.pardir): srcdir2 = sysconfig.get_config_var('srcdir') - finally: - os.chdir(cwd) self.assertEqual(srcdir, srcdir2) @unittest.skipIf(sysconfig.get_config_var('EXT_SUFFIX') is None, diff --git a/Lib/test/test_tarfile.py b/Lib/test/test_tarfile.py index 57cddb2..1412cae 100644 --- a/Lib/test/test_tarfile.py +++ b/Lib/test/test_tarfile.py @@ -1132,10 +1132,8 @@ class WriteTest(WriteTestBase, unittest.TestCase): self.assertEqual(tar.getnames(), [], "added the archive to itself") - cwd = os.getcwd() - os.chdir(TEMPDIR) - tar.add(dstname) - os.chdir(cwd) + with support.change_cwd(TEMPDIR): + tar.add(dstname) self.assertEqual(tar.getnames(), [], "added the archive to itself") finally: @@ -1292,9 +1290,7 @@ class WriteTest(WriteTestBase, unittest.TestCase): def test_cwd(self): # Test adding the current working directory. - cwd = os.getcwd() - os.chdir(TEMPDIR) - try: + with support.change_cwd(TEMPDIR): tar = tarfile.open(tmpname, self.mode) try: tar.add(".") @@ -1308,8 +1304,6 @@ class WriteTest(WriteTestBase, unittest.TestCase): self.assertTrue(t.name.startswith("./"), t.name) finally: tar.close() - finally: - os.chdir(cwd) def test_open_nonwritable_fileobj(self): for exctype in OSError, EOFError, RuntimeError: diff --git a/Lib/test/test_unicode_file.py b/Lib/test/test_unicode_file.py index faa8da3..e4709a1 100644 --- a/Lib/test/test_unicode_file.py +++ b/Lib/test/test_unicode_file.py @@ -5,7 +5,7 @@ import os, glob, time, shutil import unicodedata import unittest -from test.support import (run_unittest, rmtree, +from test.support import (run_unittest, rmtree, change_cwd, TESTFN_ENCODING, TESTFN_UNICODE, TESTFN_UNENCODABLE, create_empty_file) if not os.path.supports_unicode_filenames: @@ -82,13 +82,11 @@ class TestUnicodeFiles(unittest.TestCase): self.assertFalse(os.path.exists(filename2 + '.new')) def _do_directory(self, make_name, chdir_name): - cwd = os.getcwd() if os.path.isdir(make_name): rmtree(make_name) os.mkdir(make_name) try: - os.chdir(chdir_name) - try: + with change_cwd(chdir_name): cwd_result = os.getcwd() name_result = make_name @@ -96,8 +94,6 @@ class TestUnicodeFiles(unittest.TestCase): name_result = unicodedata.normalize("NFD", name_result) self.assertEqual(os.path.basename(cwd_result),name_result) - finally: - os.chdir(cwd) finally: os.rmdir(make_name) diff --git a/Lib/test/test_warnings/__init__.py b/Lib/test/test_warnings/__init__.py index 991a249..cea9c57 100644 --- a/Lib/test/test_warnings/__init__.py +++ b/Lib/test/test_warnings/__init__.py @@ -44,6 +44,7 @@ class BaseTest: """Basic bookkeeping required for testing.""" def setUp(self): + self.old_unittest_module = unittest.case.warnings # The __warningregistry__ needs to be in a pristine state for tests # to work properly. if '__warningregistry__' in globals(): @@ -55,10 +56,15 @@ class BaseTest: # The 'warnings' module must be explicitly set so that the proper # interaction between _warnings and 'warnings' can be controlled. sys.modules['warnings'] = self.module + # Ensure that unittest.TestCase.assertWarns() uses the same warnings + # module than warnings.catch_warnings(). Otherwise, + # warnings.catch_warnings() will be unable to remove the added filter. + unittest.case.warnings = self.module super(BaseTest, self).setUp() def tearDown(self): sys.modules['warnings'] = original_warnings + unittest.case.warnings = self.old_unittest_module super(BaseTest, self).tearDown() class PublicAPITests(BaseTest): diff --git a/Lib/test/test_wsgiref.py b/Lib/test/test_wsgiref.py index 112a1b9..8cca595 100644 --- a/Lib/test/test_wsgiref.py +++ b/Lib/test/test_wsgiref.py @@ -1,11 +1,10 @@ -from __future__ import nested_scopes # Backward compat for 2.1 from unittest import TestCase from wsgiref.util import setup_testing_defaults from wsgiref.headers import Headers from wsgiref.handlers import BaseHandler, BaseCGIHandler from wsgiref import util from wsgiref.validate import validator -from wsgiref.simple_server import WSGIServer, WSGIRequestHandler, demo_app +from wsgiref.simple_server import WSGIServer, WSGIRequestHandler from wsgiref.simple_server import make_server from io import StringIO, BytesIO, BufferedReader from socketserver import BaseServer @@ -14,8 +13,8 @@ from platform import python_implementation import os import re import sys +import unittest -from test import support class MockServer(WSGIServer): """Non-socket HTTP server""" diff --git a/Lib/test/test_zlib.py b/Lib/test/test_zlib.py index bd935ed..7cd1d7c 100644 --- a/Lib/test/test_zlib.py +++ b/Lib/test/test_zlib.py @@ -222,9 +222,9 @@ class CompressObjectTestCase(BaseCompressTestCase, unittest.TestCase): level = 2 method = zlib.DEFLATED wbits = -12 - memlevel = 9 + memLevel = 9 strategy = zlib.Z_FILTERED - co = zlib.compressobj(level, method, wbits, memlevel, strategy) + co = zlib.compressobj(level, method, wbits, memLevel, strategy) x1 = co.compress(HAMLET_SCENE) x2 = co.flush() dco = zlib.decompressobj(wbits) @@ -232,6 +232,10 @@ class CompressObjectTestCase(BaseCompressTestCase, unittest.TestCase): y2 = dco.flush() self.assertEqual(HAMLET_SCENE, y1 + y2) + # keyword arguments should also be supported + zlib.compressobj(level=level, method=method, wbits=wbits, + memLevel=memLevel, strategy=strategy, zdict=b"") + def test_compressincremental(self): # compress object in steps, decompress object as one-shot data = HAMLET_SCENE * 128 diff --git a/Lib/unittest/case.py b/Lib/unittest/case.py index 7701ad3..ac8d67d 100644 --- a/Lib/unittest/case.py +++ b/Lib/unittest/case.py @@ -583,8 +583,11 @@ class TestCase(object): finally: result.stopTest(self) return - expecting_failure = getattr(testMethod, - "__unittest_expecting_failure__", False) + expecting_failure_method = getattr(testMethod, + "__unittest_expecting_failure__", False) + expecting_failure_class = getattr(self, + "__unittest_expecting_failure__", False) + expecting_failure = expecting_failure_class or expecting_failure_method outcome = _Outcome(result) try: self._outcome = outcome @@ -1279,8 +1282,10 @@ class TestCase(object): assert expected_regex, "expected_regex must not be empty." expected_regex = re.compile(expected_regex) if not expected_regex.search(text): - msg = msg or "Regex didn't match" - msg = '%s: %r not found in %r' % (msg, expected_regex.pattern, text) + standardMsg = "Regex didn't match: %r not found in %r" % ( + expected_regex.pattern, text) + # _formatMessage ensures the longMessage option is respected + msg = self._formatMessage(msg, standardMsg) raise self.failureException(msg) def assertNotRegex(self, text, unexpected_regex, msg=None): @@ -1289,11 +1294,12 @@ class TestCase(object): unexpected_regex = re.compile(unexpected_regex) match = unexpected_regex.search(text) if match: - msg = msg or "Regex matched" - msg = '%s: %r matches %r in %r' % (msg, - text[match.start():match.end()], - unexpected_regex.pattern, - text) + standardMsg = 'Regex matched: %r matches %r in %r' % ( + text[match.start() : match.end()], + unexpected_regex.pattern, + text) + # _formatMessage ensures the longMessage option is respected + msg = self._formatMessage(msg, standardMsg) raise self.failureException(msg) @@ -1315,6 +1321,7 @@ class TestCase(object): failIf = _deprecate(assertFalse) assertRaisesRegexp = _deprecate(assertRaisesRegex) assertRegexpMatches = _deprecate(assertRegex) + assertNotRegexpMatches = _deprecate(assertNotRegex) diff --git a/Lib/unittest/mock.py b/Lib/unittest/mock.py index 4c58639..99ce1e2 100644 --- a/Lib/unittest/mock.py +++ b/Lib/unittest/mock.py @@ -2005,8 +2005,7 @@ class _Call(tuple): else: other_args = () other_kwargs = value - else: - # len 2 + elif len_other == 2: # could be (name, args) or (name, kwargs) or (args, kwargs) first, second = other if isinstance(first, str): @@ -2017,6 +2016,8 @@ class _Call(tuple): other_args, other_kwargs = (), second else: other_args, other_kwargs = first, second + else: + return False if self_name and other_name != self_name: return False diff --git a/Lib/unittest/test/test_assertions.py b/Lib/unittest/test/test_assertions.py index c349a95..e6e2bc2 100644 --- a/Lib/unittest/test/test_assertions.py +++ b/Lib/unittest/test/test_assertions.py @@ -133,7 +133,6 @@ class Test_Assertions(unittest.TestCase): try: self.assertNotRegex('Ala ma kota', r'k.t', 'Message') except self.failureException as e: - self.assertIn("'kot'", e.args[0]) self.assertIn('Message', e.args[0]) else: self.fail('assertNotRegex should have failed.') @@ -329,6 +328,20 @@ class TestLongMessage(unittest.TestCase): "^unexpectedly identical: None$", "^unexpectedly identical: None : oops$"]) + def testAssertRegex(self): + self.assertMessages('assertRegex', ('foo', 'bar'), + ["^Regex didn't match:", + "^oops$", + "^Regex didn't match:", + "^Regex didn't match: (.*) : oops$"]) + + def testAssertNotRegex(self): + self.assertMessages('assertNotRegex', ('foo', 'foo'), + ["^Regex matched:", + "^oops$", + "^Regex matched:", + "^Regex matched: (.*) : oops$"]) + def assertMessagesCM(self, methodName, args, func, errors): """ diff --git a/Lib/unittest/test/test_skipping.py b/Lib/unittest/test/test_skipping.py index 807510f..71f7b70 100644 --- a/Lib/unittest/test/test_skipping.py +++ b/Lib/unittest/test/test_skipping.py @@ -120,6 +120,39 @@ class Test_TestSkipping(unittest.TestCase): self.assertEqual(result.expectedFailures[0][0], test) self.assertTrue(result.wasSuccessful()) + def test_expected_failure_with_wrapped_class(self): + @unittest.expectedFailure + class Foo(unittest.TestCase): + def test_1(self): + self.assertTrue(False) + + events = [] + result = LoggingResult(events) + test = Foo("test_1") + test.run(result) + self.assertEqual(events, + ['startTest', 'addExpectedFailure', 'stopTest']) + self.assertEqual(result.expectedFailures[0][0], test) + self.assertTrue(result.wasSuccessful()) + + def test_expected_failure_with_wrapped_subclass(self): + class Foo(unittest.TestCase): + def test_1(self): + self.assertTrue(False) + + @unittest.expectedFailure + class Bar(Foo): + pass + + events = [] + result = LoggingResult(events) + test = Bar("test_1") + test.run(result) + self.assertEqual(events, + ['startTest', 'addExpectedFailure', 'stopTest']) + self.assertEqual(result.expectedFailures[0][0], test) + self.assertTrue(result.wasSuccessful()) + def test_expected_failure_subtests(self): # A failure in any subtest counts as the expected failure of the # whole test. diff --git a/Lib/unittest/test/testmock/testmock.py b/Lib/unittest/test/testmock/testmock.py index 97d844f..2a6069f 100644 --- a/Lib/unittest/test/testmock/testmock.py +++ b/Lib/unittest/test/testmock/testmock.py @@ -300,6 +300,9 @@ class MockTest(unittest.TestCase): self.assertEqual(mock.call_args, ((sentinel.Arg,), {"kw": sentinel.Kwarg})) + # Comparing call_args to a long sequence should not raise + # an exception. See issue 24857. + self.assertFalse(mock.call_args == "a long sequence") def test_assert_called_with(self): mock = Mock() @@ -103,6 +103,7 @@ Mike Bayer Samuel L. Bayer Donald Beaudry David Beazley +John Beck Ingolf Becker Neal Becker Robin Becker @@ -622,6 +623,7 @@ Ken Howard Brad Howes Mike Hoy Ben Hoyt +Chiu-Hsiang Hsu Chih-Hao Huang Christian Hudon Lawrence Hudson @@ -785,6 +787,7 @@ Andrew Kuchling Dave Kuhlman Jon Kuhn Toshio Kuratomi +Ilia Kurenkov Vladimir Kushnir Erno Kuusela Ross Lagerwall @@ -794,6 +797,7 @@ Thomas Lamb Valerie Lambert Jean-Baptiste "Jiba" Lamy Ronan Lamy +Peter Landry Torsten Landschoff Łukasz Langa Tino Lange @@ -827,6 +831,7 @@ Joerg Lehmann Robert Lehmann Petri Lehtinen Luke Kenneth Casson Leighton +John Leitch Tshepang Lekhonkhobe Marc-André Lemburg Mateusz Lenik @@ -911,12 +916,14 @@ Nick Mathewson Simon Mathieu Laura Matson Graham Matthews +mattip Martin Matusiak Dieter Maurer Daniel May Madison May Lucas Maystre Arnaud Mazin +Pam McA'Nulty Matt McClure Jack McCracken Rebecca McCreary @@ -1061,6 +1068,7 @@ Jan Palus Yongzhi Pan Martin Panter Mathias Panzenböck +Marco Paolini M. Papillon Peter Parente Alexandre Parenteau @@ -1121,6 +1129,7 @@ Martin Pool Iustin Pop Claudiu Popa John Popplewell +Matheus Vieira Portela Davin Potts Guillaume Pratte Florian Preinstorfer @@ -1182,6 +1191,7 @@ Vlad Riscutia Wes Rishel Daniel Riti Juan M. Bello Rivas +Mohd Sanad Zaki Rizvi Davide Rizzo Anthony Roach Carl Robben @@ -1414,6 +1424,7 @@ Eric Tiedemann July Tikhonov Tracy Tims Oren Tirosh +Tim Tisdall Jason Tishler Christian Tismer Jim Tittsler @@ -1514,6 +1525,7 @@ Bob Weiner Edward Welbourne Cliff Wells Rickard Westman +Joseph Weston Jeff Wheeler Christopher White David White @@ -2,10 +2,94 @@ Python News +++++++++++ -What's New in Python 3.5.0 final? -================================= -Release date: 2015-09-13 +What's New in Python 3.5.1 +========================== + +Release date: TBA + +Core and Builtins +----------------- + +Library +------- + +- Issue #24857: Comparing call_args to a long sequence now correctly returns a + boolean result instead of raising an exception. Patch by A Kaptur. + +- Issue #23144: Make sure that HTMLParser.feed() returns all the data, even + when convert_charrefs is True. + +- Issue #24982: shutil.make_archive() with the "zip" format now adds entries + for directories (including empty directories) in ZIP file. + +- Issue #25019: Fixed a crash caused by setting non-string key of expat parser. + Based on patch by John Leitch. + +- Issue #16180: Exit pdb if file has syntax error, instead of trapping user + in an infinite loop. Patch by Xavier de Gaye. + +- Issue #24891: Fix a race condition at Python startup if the file descriptor + of stdin (0), stdout (1) or stderr (2) is closed while Python is creating + sys.stdin, sys.stdout and sys.stderr objects. These attributes are now set + to None if the creation of the object failed, instead of raising an OSError + exception. Initial patch written by Marco Paolini. + +- Issue #24992: Fix error handling and a race condition (related to garbage + collection) in collections.OrderedDict constructor. + +- Issue #24881: Fixed setting binary mode in Python implementation of FileIO + on Windows and Cygwin. Patch from Akira Li. + +- Issue #21112: Fix regression in unittest.expectedFailure on subclasses. + Patch from Berker Peksag. + +- Issue #24764: cgi.FieldStorage.read_multi() now ignores the Content-Length + header in part headers. Patch written by Peter Landry and reviewed by Pierre + Quentel. + +- Issue #24913: Fix overrun error in deque.index(). + Found by John Leitch and Bryce Darling. + +- Issue #24774: Fix docstring in http.server.test. Patch from Chiu-Hsiang Hsu. + +- Issue #21159: Improve message in configparser.InterpolationMissingOptionError. + Patch from Łukasz Langa. + +- Issue #20362: Honour TestCase.longMessage correctly in assertRegex. + Patch from Ilia Kurenkov. + +- Issue #23572: Fixed functools.singledispatch on classes with falsy + metaclasses. Patch by Ethan Furman. + +Documentation +------------- + +- Issue #24952: Clarify the default size argument of stack_size() in + the "threading" and "_thread" modules. Patch from Mattip. + +- Issue #23725: Overhaul tempfile docs. Note deprecated status of mktemp. + Patch from Zbigniew Jędrzejewski-Szmek. + +- Issue #24808: Update the types of some PyTypeObject fields. Patch by + Joseph Weston. + +- Issue #22812: Fix unittest discovery examples. + Patch from Pam McA'Nulty. + +Tests +----- + +- PCbuild\rt.bat now accepts an unlimited number of arguments to pass along + to regrtest.py. Previously there was a limit of 9. + +Build +----- + +- Issue #24910: Windows MSIs now have unique display names. + +- Issue #24986: It is now possible to build Python on Windows without errors + when external libraries are not available. What's New in Python 3.5.0 release candidate 4? diff --git a/Misc/Porting b/Misc/Porting index c283cb2..c43b112 100644 --- a/Misc/Porting +++ b/Misc/Porting @@ -1,41 +1 @@ -Q. I want to port Python to a new platform. How do I begin? - -A. I guess the two things to start with is to familiarize yourself -with are the development system for your target platform and the -generic build process for Python. Make sure you can compile and run a -simple hello-world program on your target platform. Make sure you can -compile and run the Python interpreter on a platform to which it has -already been ported (preferably Unix, but Mac or Windows will do, -too). - -I also would never start something like this without at least -medium-level understanding of your target platform (i.e. how it is -generally used, how to write platform specific apps etc.) and Python -(or else you'll never know how to test the results). - -The build process for Python, in particular the Makefiles in the -source distribution, will give you a hint on which files to compile -for Python. Not all source files are relevant -- some are platform -specific, others are only used in emergencies (e.g. getopt.c). The -Makefiles tell the story. - -You'll also need a pyconfig.h file tailored for your platform. You can -start with pyconfig.h.in, read the comments and turn on definitions that -apply to your platform. - -And you'll need a config.c file, which lists the built-in modules you -support. Start with Modules/config.c.in. - -Finally, you'll run into some things that aren't supported on your -target platform. Forget about the posix module for now -- simply take -it out of the config.c file. - -Bang on it until you get a >>> prompt. (You may have to disable the -importing of "site.py" by passing the -S option.) - -Then bang on it until it executes very simple Python statements. - -Now bang on it some more. At some point you'll want to use the os -module; this is the time to start thinking about what to do with the -posix module. It's okay to simply #ifdef out those functions that -cause problems; the remaining ones will be quite useful. +This document is moved to https://docs.python.org/devguide/faq.html#how-do-i-port-python-to-a-new-platform diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c index 3856d83..f450f25 100644 --- a/Modules/_collectionsmodule.c +++ b/Modules/_collectionsmodule.c @@ -419,9 +419,11 @@ deque_extend(dequeobject *deque, PyObject *iterable) deque->rightblock->data[deque->rightindex] = item; deque_trim_left(deque); } - Py_DECREF(it); - if (PyErr_Occurred()) + if (PyErr_Occurred()) { + Py_DECREF(it); return NULL; + } + Py_DECREF(it); Py_RETURN_NONE; } @@ -480,9 +482,11 @@ deque_extendleft(dequeobject *deque, PyObject *iterable) deque->leftblock->data[deque->leftindex] = item; deque_trim_right(deque); } - Py_DECREF(it); - if (PyErr_Occurred()) + if (PyErr_Occurred()) { + Py_DECREF(it); return NULL; + } + Py_DECREF(it); Py_RETURN_NONE; } @@ -497,8 +501,8 @@ deque_inplace_concat(dequeobject *deque, PyObject *other) result = deque_extend(deque, other); if (result == NULL) return result; - Py_DECREF(result); Py_INCREF(deque); + Py_DECREF(result); return (PyObject *)deque; } @@ -1260,8 +1264,8 @@ deque_repr(PyObject *deque) aslist, ((dequeobject *)deque)->maxlen); else result = PyUnicode_FromFormat("deque(%R)", aslist); - Py_DECREF(aslist); Py_ReprLeave(deque); + Py_DECREF(aslist); return result; } diff --git a/Modules/_decimal/libmpdec/mpdecimal.c b/Modules/_decimal/libmpdec/mpdecimal.c index 21d2222..593f9f5 100644 --- a/Modules/_decimal/libmpdec/mpdecimal.c +++ b/Modules/_decimal/libmpdec/mpdecimal.c @@ -43,6 +43,7 @@ #ifdef PPRO #if defined(_MSC_VER) #include <float.h> + #pragma float_control(precise, on) #pragma fenv_access(on) #elif !defined(__OpenBSD__) && !defined(__NetBSD__) /* C99 */ diff --git a/Modules/clinic/posixmodule.c.h b/Modules/clinic/posixmodule.c.h index 98eeae4..9ef702a 100644 --- a/Modules/clinic/posixmodule.c.h +++ b/Modules/clinic/posixmodule.c.h @@ -1487,10 +1487,10 @@ PyDoc_STRVAR(os_utime__doc__, "\n" "If times is not None, it must be a tuple (atime, mtime);\n" " atime and mtime should be expressed as float seconds since the epoch.\n" -"If ns is not None, it must be a tuple (atime_ns, mtime_ns);\n" +"If ns is specified, it must be a tuple (atime_ns, mtime_ns);\n" " atime_ns and mtime_ns should be expressed as integer nanoseconds\n" " since the epoch.\n" -"If both times and ns are None, utime uses the current time.\n" +"If times is None and ns is unspecified, utime uses the current time.\n" "Specifying tuples for both times and ns is an error.\n" "\n" "If dir_fd is not None, it should be a file descriptor open to a directory,\n" @@ -5788,4 +5788,4 @@ exit: #ifndef OS_SET_HANDLE_INHERITABLE_METHODDEF #define OS_SET_HANDLE_INHERITABLE_METHODDEF #endif /* !defined(OS_SET_HANDLE_INHERITABLE_METHODDEF) */ -/*[clinic end generated code: output=f3f92b2d2e2c3fe3 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=95824c52fd034654 input=a9049054013a1b77]*/ diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c index ec8c526..c0baf6a 100644 --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -4560,9 +4560,7 @@ typedef struct { } \ -#define UTIME_HAVE_DIR_FD (defined(HAVE_FUTIMESAT) || defined(HAVE_UTIMENSAT)) - -#if UTIME_HAVE_DIR_FD +#if defined(HAVE_FUTIMESAT) || defined(HAVE_UTIMENSAT) static int utime_dir_fd(utime_t *ut, int dir_fd, char *path, int follow_symlinks) @@ -4588,9 +4586,7 @@ utime_dir_fd(utime_t *ut, int dir_fd, char *path, int follow_symlinks) #define FUTIMENSAT_DIR_FD_CONVERTER dir_fd_unavailable #endif -#define UTIME_HAVE_FD (defined(HAVE_FUTIMES) || defined(HAVE_FUTIMENS)) - -#if UTIME_HAVE_FD +#if defined(HAVE_FUTIMES) || defined(HAVE_FUTIMENS) static int utime_fd(utime_t *ut, int fd) @@ -4683,7 +4679,7 @@ os.utime dir_fd: dir_fd(requires='futimensat') = None follow_symlinks: bool=True -# "utime(path, times=None, *, ns=None, dir_fd=None, follow_symlinks=True)\n\ +# "utime(path, times=None, *[, ns], dir_fd=None, follow_symlinks=True)\n\ Set the access and modified time of path. @@ -4693,10 +4689,10 @@ On some platforms, path may also be specified as an open file descriptor. If times is not None, it must be a tuple (atime, mtime); atime and mtime should be expressed as float seconds since the epoch. -If ns is not None, it must be a tuple (atime_ns, mtime_ns); +If ns is specified, it must be a tuple (atime_ns, mtime_ns); atime_ns and mtime_ns should be expressed as integer nanoseconds since the epoch. -If both times and ns are None, utime uses the current time. +If times is None and ns is unspecified, utime uses the current time. Specifying tuples for both times and ns is an error. If dir_fd is not None, it should be a file descriptor open to a directory, @@ -4714,7 +4710,7 @@ dir_fd and follow_symlinks may not be available on your platform. static PyObject * os_utime_impl(PyModuleDef *module, path_t *path, PyObject *times, PyObject *ns, int dir_fd, int follow_symlinks) -/*[clinic end generated code: output=31f3434e560ba2f0 input=1f18c17d5941aa82]*/ +/*[clinic end generated code: output=31f3434e560ba2f0 input=081cdc54ca685385]*/ { #ifdef MS_WINDOWS HANDLE hFile; @@ -4835,13 +4831,13 @@ os_utime_impl(PyModuleDef *module, path_t *path, PyObject *times, else #endif -#if UTIME_HAVE_DIR_FD +#if defined(HAVE_FUTIMESAT) || defined(HAVE_UTIMENSAT) if ((dir_fd != DEFAULT_DIR_FD) || (!follow_symlinks)) result = utime_dir_fd(&utime, dir_fd, path->narrow, follow_symlinks); else #endif -#if UTIME_HAVE_FD +#if defined(HAVE_FUTIMES) || defined(HAVE_FUTIMENS) if (path->fd != -1) result = utime_fd(&utime, path->fd); else @@ -8239,10 +8235,10 @@ os_write_impl(PyModuleDef *module, int fd, Py_buffer *data) #ifdef HAVE_SENDFILE PyDoc_STRVAR(posix_sendfile__doc__, -"sendfile(out, in, offset, nbytes) -> byteswritten\n\ -sendfile(out, in, offset, nbytes, headers=None, trailers=None, flags=0)\n\ +"sendfile(out, in, offset, count) -> byteswritten\n\ +sendfile(out, in, offset, count[, headers][, trailers], flags=0)\n\ -> byteswritten\n\ -Copy nbytes bytes from file descriptor in to file descriptor out."); +Copy count bytes from file descriptor in to file descriptor out."); /* AC 3.5: don't bother converting, has optional group*/ static PyObject * @@ -8262,6 +8258,7 @@ posix_sendfile(PyObject *self, PyObject *args, PyObject *kwdict) off_t sbytes; struct sf_hdtr sf; int flags = 0; + /* Beware that "in" clashes with Python's own "in" operator keyword */ static char *keywords[] = {"out", "in", "offset", "count", "headers", "trailers", "flags", NULL}; @@ -8281,7 +8278,7 @@ posix_sendfile(PyObject *self, PyObject *args, PyObject *kwdict) if (headers != NULL) { if (!PySequence_Check(headers)) { PyErr_SetString(PyExc_TypeError, - "sendfile() headers must be a sequence or None"); + "sendfile() headers must be a sequence"); return NULL; } else { Py_ssize_t i = 0; /* Avoid uninitialized warning */ @@ -8298,7 +8295,7 @@ posix_sendfile(PyObject *self, PyObject *args, PyObject *kwdict) if (trailers != NULL) { if (!PySequence_Check(trailers)) { PyErr_SetString(PyExc_TypeError, - "sendfile() trailers must be a sequence or None"); + "sendfile() trailers must be a sequence"); return NULL; } else { Py_ssize_t i = 0; /* Avoid uninitialized warning */ diff --git a/Modules/pyexpat.c b/Modules/pyexpat.c index a43887e..9a6da73 100644 --- a/Modules/pyexpat.c +++ b/Modules/pyexpat.c @@ -1378,11 +1378,16 @@ static int xmlparse_setattro(xmlparseobject *self, PyObject *name, PyObject *v) { /* Set attribute 'name' to value 'v'. v==NULL means delete */ + if (!PyUnicode_Check(name)) { + PyErr_Format(PyExc_TypeError, + "attribute name must be string, not '%.200s'", + name->ob_type->tp_name); + return -1; + } if (v == NULL) { PyErr_SetString(PyExc_RuntimeError, "Cannot delete attribute"); return -1; } - assert(PyUnicode_Check(name)); if (PyUnicode_CompareWithASCIIString(name, "buffer_text") == 0) { int b = PyObject_IsTrue(v); if (b < 0) @@ -1435,17 +1440,18 @@ xmlparse_setattro(xmlparseobject *self, PyObject *name, PyObject *v) return -1; } - new_buffer_size=PyLong_AS_LONG(v); + new_buffer_size = PyLong_AsLong(v); + if (new_buffer_size <= 0) { + if (!PyErr_Occurred()) + PyErr_SetString(PyExc_ValueError, "buffer_size must be greater than zero"); + return -1; + } + /* trivial case -- no change */ if (new_buffer_size == self->buffer_size) { return 0; } - if (new_buffer_size <= 0) { - PyErr_SetString(PyExc_ValueError, "buffer_size must be greater than zero"); - return -1; - } - /* check maximum */ if (new_buffer_size > INT_MAX) { char errmsg[100]; diff --git a/Objects/odictobject.c b/Objects/odictobject.c index 7dbaa89..104d6c1 100644 --- a/Objects/odictobject.c +++ b/Objects/odictobject.c @@ -98,7 +98,6 @@ For removing nodes: Others: -* _odict_initialize(od) * _odict_find_node(od, key) * _odict_keys_equal(od1, od2) @@ -602,15 +601,6 @@ _odict_get_index(PyODictObject *od, PyObject *key) return _odict_get_index_hash(od, key, hash); } -static int -_odict_initialize(PyODictObject *od) -{ - od->od_state = 0; - _odict_FIRST(od) = NULL; - _odict_LAST(od) = NULL; - return _odict_resize((PyODictObject *)od); -} - /* Returns NULL if there was some error or the key was not found. */ static _ODictNode * _odict_find_node(PyODictObject *od, PyObject *key) @@ -744,7 +734,7 @@ _odict_pop_node(PyODictObject *od, _ODictNode *node, PyObject *key) /* If someone calls PyDict_DelItem() directly on an OrderedDict, we'll get all sorts of problems here. In PyODict_DelItem we make sure to call _odict_clear_node first. - + This matters in the case of colliding keys. Suppose we add 3 keys: [A, B, C], where the hash of C collides with A and the next possible index in the hash table is occupied by B. If we remove B then for C @@ -1739,14 +1729,28 @@ odict_init(PyObject *self, PyObject *args, PyObject *kwds) static PyObject * odict_new(PyTypeObject *type, PyObject *args, PyObject *kwds) { - PyObject *od = PyDict_Type.tp_new(type, args, kwds); - if (od != NULL) { - if (_odict_initialize((PyODictObject *)od) < 0) - return NULL; - ((PyODictObject *)od)->od_inst_dict = PyDict_New(); - ((PyODictObject *)od)->od_weakreflist = NULL; + PyObject *dict; + PyODictObject *od; + + dict = PyDict_New(); + if (dict == NULL) + return NULL; + + od = (PyODictObject *)PyDict_Type.tp_new(type, args, kwds); + if (od == NULL) { + Py_DECREF(dict); + return NULL; } - return od; + + od->od_inst_dict = dict; + /* type constructor fills the memory with zeros (see + PyType_GenericAlloc()), there is no need to set them to zero again */ + if (_odict_resize(od) < 0) { + Py_DECREF(od); + return NULL; + } + + return (PyObject*)od; } /* PyODict_Type */ diff --git a/PC/pyconfig.h b/PC/pyconfig.h index 324a130..d44173a 100644 --- a/PC/pyconfig.h +++ b/PC/pyconfig.h @@ -147,7 +147,11 @@ WIN32 is still required for the locale module. #define MS_WINI64 #define PYD_PLATFORM_TAG "win_ia64" #elif defined(_M_X64) || defined(_M_AMD64) +#if defined(__INTEL_COMPILER) +#define COMPILER ("[ICC v." _Py_STRINGIZE(__INTEL_COMPILER) " 64 bit (amd64) with MSC v." _Py_STRINGIZE(_MSC_VER) " CRT]") +#else #define COMPILER _Py_PASTE_VERSION("64 bit (AMD64)") +#endif /* __INTEL_COMPILER */ #define MS_WINX64 #define PYD_PLATFORM_TAG "win_amd64" #else @@ -194,7 +198,11 @@ typedef _W64 int ssize_t; #if defined(MS_WIN32) && !defined(MS_WIN64) #if defined(_M_IX86) +#if defined(__INTEL_COMPILER) +#define COMPILER ("[ICC v." _Py_STRINGIZE(__INTEL_COMPILER) " 32 bit (Intel) with MSC v." _Py_STRINGIZE(_MSC_VER) " CRT]") +#else #define COMPILER _Py_PASTE_VERSION("32 bit (Intel)") +#endif /* __INTEL_COMPILER */ #define PYD_PLATFORM_TAG "win32" #elif defined(_M_ARM) #define COMPILER _Py_PASTE_VERSION("32 bit (ARM)") diff --git a/PCbuild/build.bat b/PCbuild/build.bat index 0fb01d7..2c41fb2 100644 --- a/PCbuild/build.bat +++ b/PCbuild/build.bat @@ -1,19 +1,46 @@ @echo off
-rem A batch program to build or rebuild a particular configuration,
-rem just for convenience.
-
-rem Arguments:
-rem -c Set the configuration (default: Release)
-rem -p Set the platform (x64 or Win32, default: Win32)
-rem -r Target Rebuild instead of Build
-rem -t Set the target manually (Build, Rebuild, Clean, or CleanAll)
-rem -d Set the configuration to Debug
-rem -e Pull in external libraries using get_externals.bat
-rem -m Enable parallel build (enabled by default)
-rem -M Disable parallel build
-rem -v Increased output messages
-rem -k Attempt to kill any running Pythons before building (usually unnecessary)
+goto Run
+:Usage
+echo.%~nx0 [flags and arguments] [quoted MSBuild options]
+echo.
+echo.Build CPython from the command line. Requires the appropriate
+echo.version(s) of Microsoft Visual Studio to be installed (see readme.txt).
+echo.Also requires Subversion (svn.exe) to be on PATH if the '-e' flag is
+echo.given.
+echo.
+echo.After the flags recognized by this script, up to 9 arguments to be passed
+echo.directly to MSBuild may be passed. If the argument contains an '=', the
+echo.entire argument must be quoted (e.g. `%~nx0 "/p:PlatformToolset=v100"`)
+echo.
+echo.Available flags:
+echo. -h Display this help message
+echo. -V Display version information for the current build
+echo. -r Target Rebuild instead of Build
+echo. -d Set the configuration to Debug
+echo. -e Build external libraries fetched by get_externals.bat
+echo. Extension modules that depend on external libraries will not attempt
+echo. to build if this flag is not present
+echo. -m Enable parallel build (enabled by default)
+echo. -M Disable parallel build
+echo. -v Increased output messages
+echo. -k Attempt to kill any running Pythons before building (usually done
+echo. automatically by the pythoncore project)
+echo.
+echo.Available flags to avoid building certain modules.
+echo.These flags have no effect if '-e' is not given:
+echo. --no-ssl Do not attempt to build _ssl
+echo. --no-tkinter Do not attempt to build Tkinter
+echo.
+echo.Available arguments:
+echo. -c Release ^| Debug ^| PGInstrument ^| PGUpdate
+echo. Set the configuration (default: Release)
+echo. -p x64 ^| Win32
+echo. Set the platform (default: Win32)
+echo. -t Build ^| Rebuild ^| Clean ^| CleanAll
+echo. Set the target manually
+exit /b 127
+:Run
setlocal
set platf=Win32
set vs_platf=x86
@@ -25,17 +52,29 @@ set verbose=/nologo /v:m set kill=
:CheckOpts
+if "%~1"=="-h" goto Usage
if "%~1"=="-c" (set conf=%2) & shift & shift & goto CheckOpts
if "%~1"=="-p" (set platf=%2) & shift & shift & goto CheckOpts
if "%~1"=="-r" (set target=Rebuild) & shift & goto CheckOpts
if "%~1"=="-t" (set target=%2) & shift & shift & goto CheckOpts
if "%~1"=="-d" (set conf=Debug) & shift & goto CheckOpts
-if "%~1"=="-e" call "%dir%get_externals.bat" & shift & goto CheckOpts
if "%~1"=="-m" (set parallel=/m) & shift & goto CheckOpts
if "%~1"=="-M" (set parallel=) & shift & goto CheckOpts
if "%~1"=="-v" (set verbose=/v:n) & shift & goto CheckOpts
if "%~1"=="-k" (set kill=true) & shift & goto CheckOpts
if "%~1"=="-V" shift & goto Version
+rem These use the actual property names used by MSBuild. We could just let
+rem them in through the environment, but we specify them on the command line
+rem anyway for visibility so set defaults after this
+if "%~1"=="-e" (set IncludeExternals=true) & shift & goto CheckOpts
+if "%~1"=="--no-ssl" (set IncludeSSL=false) & shift & goto CheckOpts
+if "%~1"=="--no-tkinter" (set IncludeTkinter=false) & shift & goto CheckOpts
+
+if "%IncludeExternals%"=="" set IncludeExternals=false
+if "%IncludeSSL%"=="" set IncludeSSL=true
+if "%IncludeTkinter%"=="" set IncludeTkinter=true
+
+if "%IncludeExternals%"=="true" call "%dir%get_externals.bat"
if "%platf%"=="x64" (set vs_platf=x86_amd64)
@@ -43,14 +82,18 @@ rem Setup the environment call "%dir%env.bat" %vs_platf% >nul
if "%kill%"=="true" (
- msbuild /v:m /nologo /target:KillPython "%pcbuild%\pythoncore.vcxproj" /p:Configuration=%conf% /p:Platform=%platf% /p:KillPython=true
+ msbuild /v:m /nologo /target:KillPython "%dir%\pythoncore.vcxproj" /p:Configuration=%conf% /p:Platform=%platf% /p:KillPython=true
)
rem Call on MSBuild to do the work, echo the command.
rem Passing %1-9 is not the preferred option, but argument parsing in
rem batch is, shall we say, "lackluster"
echo on
-msbuild "%dir%pcbuild.proj" /t:%target% %parallel% %verbose% /p:Configuration=%conf% /p:Platform=%platf% %1 %2 %3 %4 %5 %6 %7 %8 %9
+msbuild "%dir%pcbuild.proj" /t:%target% %parallel% %verbose%^
+ /p:Configuration=%conf% /p:Platform=%platf%^
+ /p:IncludeExternals=%IncludeExternals%^
+ /p:IncludeSSL=%IncludeSSL% /p:IncludeTkinter=%IncludeTkinter%^
+ %1 %2 %3 %4 %5 %6 %7 %8 %9
@goto :eof
diff --git a/PCbuild/get_externals.bat b/PCbuild/get_externals.bat index f9b740f..0b3c08b 100644 --- a/PCbuild/get_externals.bat +++ b/PCbuild/get_externals.bat @@ -51,16 +51,17 @@ if ERRORLEVEL 9009 ( echo.Fetching external libraries...
-for %%e in (
- bzip2-1.0.6
- nasm-2.11.06
- openssl-1.0.2d
- tcl-core-8.6.4.2
- tk-8.6.4.2
- tix-8.4.3.6
- sqlite-3.8.11.0
- xz-5.0.5
- ) do (
+set libraries=
+set libraries=%libraries% bzip2-1.0.6
+if NOT "%IncludeSSL%"=="false" set libraries=%libraries% nasm-2.11.06
+if NOT "%IncludeSSL%"=="false" set libraries=%libraries% openssl-1.0.2d
+set libraries=%libraries% sqlite-3.8.11.0
+if NOT "%IncludeTkinter%"=="false" set libraries=%libraries% tcl-core-8.6.4.2
+if NOT "%IncludeTkinter%"=="false" set libraries=%libraries% tk-8.6.4.2
+if NOT "%IncludeTkinter%"=="false" set libraries=%libraries% tix-8.4.3.6
+set libraries=%libraries% xz-5.0.5
+
+for %%e in (%libraries%) do (
if exist %%e (
echo.%%e already exists, skipping.
) else (
diff --git a/PCbuild/pcbuild.proj b/PCbuild/pcbuild.proj index 813831e..36621c9 100644 --- a/PCbuild/pcbuild.proj +++ b/PCbuild/pcbuild.proj @@ -5,8 +5,10 @@ <Platform Condition="'$(Platform)' == ''">Win32</Platform> <Configuration Condition="'$(Configuration)' == ''">Release</Configuration> <IncludeExtensions Condition="'$(IncludeExtensions)' == ''">true</IncludeExtensions> + <IncludeExternals Condition="'$(IncludeExternals)' == ''">true</IncludeExternals> <IncludeTests Condition="'$(IncludeTest)' == ''">true</IncludeTests> <IncludeSSL Condition="'$(IncludeSSL)' == ''">true</IncludeSSL> + <IncludeTkinter Condition="'$(IncludeTkinter)' == ''">true</IncludeTkinter> </PropertyGroup> <ItemDefinitionGroup> @@ -25,7 +27,7 @@ <!-- Parallel build is explicitly disabled for this project because it causes many conflicts between pythoncore and projects that depend - in pythoncore. Once the core DLL has been built, subsequent + on pythoncore. Once the core DLL has been built, subsequent projects will be built in parallel. --> <Projects Include="pythoncore.vcxproj"> @@ -40,10 +42,14 @@ <!-- _freeze_importlib --> <Projects Include="_freeze_importlib.vcxproj" /> <!-- Extension modules --> - <ExtensionModules Include="_bz2;_ctypes;_decimal;_elementtree;_lzma;_msi;_multiprocessing;_overlapped;_sqlite3;_tkinter;tix;pyexpat;select;unicodedata;winsound" /> + <ExtensionModules Include="_ctypes;_decimal;_elementtree;_msi;_multiprocessing;_overlapped;pyexpat;select;unicodedata;winsound" /> + <!-- Extension modules that require external sources --> + <ExternalModules Include="_bz2;_lzma;_sqlite3" /> <!-- _ssl will build _socket as well, which may cause conflicts in parallel builds --> - <ExtensionModules Include="_socket" Condition="!$(IncludeSSL)" /> - <ExtensionModules Include="_ssl;_hashlib" Condition="$(IncludeSSL)" /> + <ExtensionModules Include="_socket" Condition="!$(IncludeSSL) or !$(IncludeExternals)" /> + <ExternalModules Include="_ssl;_hashlib" Condition="$(IncludeSSL)" /> + <ExternalModules Include="_tkinter;tix" Condition="$(IncludeTkinter)" /> + <ExtensionModules Include="@(ExternalModules->'%(Identity)')" Condition="$(IncludeExternals)" /> <Projects Include="@(ExtensionModules->'%(Identity).vcxproj')" Condition="$(IncludeExtensions)" /> <!-- Test modules --> <TestModules Include="_ctypes_test;_testbuffer;_testcapi;_testembed;_testimportmultiple;_testmultiphase" /> diff --git a/PCbuild/readme.txt b/PCbuild/readme.txt index 68cdb0f..7e7ab64 100644 --- a/PCbuild/readme.txt +++ b/PCbuild/readme.txt @@ -41,8 +41,7 @@ The solution currently supports two platforms. The Win32 platform is used to build standard x86-compatible 32-bit binaries, output into the
win32 sub-directory. The x64 platform is used for building 64-bit AMD64
(aka x86_64 or EM64T) binaries, output into the amd64 sub-directory.
-The Itanium (IA-64) platform is no longer supported. See the "Building
-for AMD64" section below for more information about 64-bit builds.
+The Itanium (IA-64) platform is no longer supported.
Four configuration options are supported by the solution:
Debug
@@ -76,29 +75,7 @@ officially supported. By default, build.bat will build Python in Release configuration for
the 32-bit Win32 platform. It accepts several arguments to change
-this behavior:
-
- -c <configuration> Set the configuration (see above)
- -d Shortcut for "-c Debug"
- -p <platform> Set the platform to build for ("Win32" or "x64")
- -r Rebuild instead of just building
- -t <target> Set the target (Build, Rebuild, Clean or CleanAll)
- -e Use get_externals.bat to fetch external sources
- -M Don't build in parallel
- -v Increased output messages
-
-Up to 9 MSBuild switches can also be passed, though they must be passed
-after specifying any of the above switches. For example, use:
-
- build.bat -e -d /fl
-
-to do a debug build with externals fetched as needed and write detailed
-build logs to a file. If the MSBuild switch requires an equal sign
-("="), the entire switch must be quoted:
-
- build.bat -e -d "/p:ExternalsDir=P:\cpython-externals"
-
-There may also be other situations where quotes are necessary.
+this behavior, try `build.bat -h` to learn more.
C Runtime
@@ -129,8 +106,6 @@ pythoncore .dll and .lib
python
.exe
-make_buildinfo, make_versioninfo
- helpers to provide necessary information to the build process
These sub-projects provide extra executables that are useful for running
CPython in different ways:
@@ -152,9 +127,6 @@ categories: _freeze_importlib
_freeze_importlib.exe, used to regenerate Python\importlib.h after
changes have been made to Lib\importlib\_bootstrap.py
-bdist_wininst
- ..\Lib\distutils\command\wininst-14.0[-amd64].exe, the base
- executable used by the distutils bdist_wininst command
python3dll
python3.dll, the PEP 384 Stable ABI dll
xxlimited
@@ -274,14 +246,8 @@ as the values of certain properties in order for the build solution to find them. This is an advanced topic and not necessarily fully
supported.
-
-Building for AMD64
-------------------
-
-The build process for AMD64 / x64 is very similar to standard builds,
-you just have to set x64 as platform. In addition, the HOST_PYTHON
-environment variable must point to a Python interpreter (at least 2.4),
-to support cross-compilation from Win32.
+The get_externals.bat script is called automatically by build.bat when
+you pass the '-e' option to it.
Profile Guided Optimization
@@ -298,7 +264,7 @@ It creates the PGI files, runs the unit test suite or PyBench with the PGI python, and finally creates the optimized files.
See
- http://msdn.microsoft.com/en-us/library/e7k32f4k(VS.100).aspx
+ http://msdn.microsoft.com/en-us/library/e7k32f4k(VS.140).aspx
for more on this topic.
@@ -338,5 +304,5 @@ Your Own Extension DLLs -----------------------
If you want to create your own extension module DLL (.pyd), there's an
-example with easy-to-follow instructions in ..\PC\example\; read the
+example with easy-to-follow instructions in ..\PC\example_nt\; read the
file readme.txt there first.
diff --git a/PCbuild/rt.bat b/PCbuild/rt.bat index 039c810..2d93b80 100644 --- a/PCbuild/rt.bat +++ b/PCbuild/rt.bat @@ -32,15 +32,17 @@ set prefix=%pcbuild%win32\ set suffix=
set qmode=
set dashO=
+set regrtestargs=
:CheckOpts
if "%1"=="-O" (set dashO=-O) & shift & goto CheckOpts
if "%1"=="-q" (set qmode=yes) & shift & goto CheckOpts
if "%1"=="-d" (set suffix=_d) & shift & goto CheckOpts
if "%1"=="-x64" (set prefix=%pcbuild%amd64\) & shift & goto CheckOpts
+if NOT "%1"=="" (set regrtestargs=%regrtestargs% %1) & shift & goto CheckOpts
set exe=%prefix%python%suffix%.exe
-set cmd="%exe%" %dashO% -Wd -E -bb "%pcbuild%..\lib\test\regrtest.py" %1 %2 %3 %4 %5 %6 %7 %8 %9
+set cmd="%exe%" %dashO% -Wd -E -bb "%pcbuild%..\lib\test\regrtest.py" %regrtestargs%
if defined qmode goto Qmode
echo Deleting .pyc/.pyo files ...
diff --git a/Python/codecs.c b/Python/codecs.c index 596bd80..d90bf73 100644 --- a/Python/codecs.c +++ b/Python/codecs.c @@ -966,7 +966,6 @@ PyObject *PyCodec_BackslashReplaceErrors(PyObject *exc) } static _PyUnicode_Name_CAPI *ucnhash_CAPI = NULL; -static int ucnhash_initialized = 0; PyObject *PyCodec_NameReplaceErrors(PyObject *exc) { @@ -988,17 +987,17 @@ PyObject *PyCodec_NameReplaceErrors(PyObject *exc) return NULL; if (!(object = PyUnicodeEncodeError_GetObject(exc))) return NULL; - if (!ucnhash_initialized) { + if (!ucnhash_CAPI) { /* load the unicode data module */ ucnhash_CAPI = (_PyUnicode_Name_CAPI *)PyCapsule_Import( PyUnicodeData_CAPSULE_NAME, 1); - ucnhash_initialized = 1; + if (!ucnhash_CAPI) + return NULL; } for (i = start, ressize = 0; i < end; ++i) { /* object is guaranteed to be "ready" */ c = PyUnicode_READ_CHAR(object, i); - if (ucnhash_CAPI && - ucnhash_CAPI->getname(NULL, c, buffer, sizeof(buffer), 1)) { + if (ucnhash_CAPI->getname(NULL, c, buffer, sizeof(buffer), 1)) { replsize = 1+1+1+(int)strlen(buffer)+1; } else if (c >= 0x10000) { @@ -1021,8 +1020,7 @@ PyObject *PyCodec_NameReplaceErrors(PyObject *exc) i < end; ++i) { c = PyUnicode_READ_CHAR(object, i); *outp++ = '\\'; - if (ucnhash_CAPI && - ucnhash_CAPI->getname(NULL, c, buffer, sizeof(buffer), 1)) { + if (ucnhash_CAPI->getname(NULL, c, buffer, sizeof(buffer), 1)) { *outp++ = 'N'; *outp++ = '{'; strcpy((char *)outp, buffer); diff --git a/Python/pylifecycle.c b/Python/pylifecycle.c index a17adf7..df66fa0 100644 --- a/Python/pylifecycle.c +++ b/Python/pylifecycle.c @@ -1,4 +1,3 @@ - /* Python interpreter top-level routines, including init/exit */ #include "Python.h" @@ -963,6 +962,23 @@ initsite(void) } } +/* Check if a file descriptor is valid or not. + Return 0 if the file descriptor is invalid, return non-zero otherwise. */ +static int +is_valid_fd(int fd) +{ + int fd2; + if (fd < 0 || !_PyVerify_fd(fd)) + return 0; + _Py_BEGIN_SUPPRESS_IPH + fd2 = dup(fd); + if (fd2 >= 0) + close(fd2); + _Py_END_SUPPRESS_IPH + return fd2 >= 0; +} + +/* returns Py_None if the fd is not valid */ static PyObject* create_stdio(PyObject* io, int fd, int write_mode, char* name, @@ -978,6 +994,9 @@ create_stdio(PyObject* io, _Py_IDENTIFIER(TextIOWrapper); _Py_IDENTIFIER(mode); + if (!is_valid_fd(fd)) + Py_RETURN_NONE; + /* stdin is always opened in buffered mode, first because it shouldn't make a difference in common use cases, second because TextIOWrapper depends on the presence of a read1() method which only exists on @@ -1059,21 +1078,15 @@ error: Py_XDECREF(stream); Py_XDECREF(text); Py_XDECREF(raw); - return NULL; -} -static int -is_valid_fd(int fd) -{ - int dummy_fd; - if (fd < 0 || !_PyVerify_fd(fd)) - return 0; - _Py_BEGIN_SUPPRESS_IPH - dummy_fd = dup(fd); - if (dummy_fd >= 0) - close(dummy_fd); - _Py_END_SUPPRESS_IPH - return dummy_fd >= 0; + if (PyErr_ExceptionMatches(PyExc_OSError) && !is_valid_fd(fd)) { + /* Issue #24891: the file descriptor was closed after the first + is_valid_fd() check was called. Ignore the OSError and set the + stream to None. */ + PyErr_Clear(); + Py_RETURN_NONE; + } + return NULL; } /* Initialize sys.stdin, stdout, stderr and builtins.open */ @@ -1158,30 +1171,18 @@ initstdio(void) * and fileno() may point to an invalid file descriptor. For example * GUI apps don't have valid standard streams by default. */ - if (!is_valid_fd(fd)) { - std = Py_None; - Py_INCREF(std); - } - else { - std = create_stdio(iomod, fd, 0, "<stdin>", encoding, errors); - if (std == NULL) - goto error; - } /* if (fd < 0) */ + std = create_stdio(iomod, fd, 0, "<stdin>", encoding, errors); + if (std == NULL) + goto error; PySys_SetObject("__stdin__", std); _PySys_SetObjectId(&PyId_stdin, std); Py_DECREF(std); /* Set sys.stdout */ fd = fileno(stdout); - if (!is_valid_fd(fd)) { - std = Py_None; - Py_INCREF(std); - } - else { - std = create_stdio(iomod, fd, 1, "<stdout>", encoding, errors); - if (std == NULL) - goto error; - } /* if (fd < 0) */ + std = create_stdio(iomod, fd, 1, "<stdout>", encoding, errors); + if (std == NULL) + goto error; PySys_SetObject("__stdout__", std); _PySys_SetObjectId(&PyId_stdout, std); Py_DECREF(std); @@ -1189,15 +1190,9 @@ initstdio(void) #if 1 /* Disable this if you have trouble debugging bootstrap stuff */ /* Set sys.stderr, replaces the preliminary stderr */ fd = fileno(stderr); - if (!is_valid_fd(fd)) { - std = Py_None; - Py_INCREF(std); - } - else { - std = create_stdio(iomod, fd, 1, "<stderr>", encoding, "backslashreplace"); - if (std == NULL) - goto error; - } /* if (fd < 0) */ + std = create_stdio(iomod, fd, 1, "<stderr>", encoding, "backslashreplace"); + if (std == NULL) + goto error; /* Same as hack above, pre-import stderr's codec to avoid recursion when import.c tries to write to stderr in verbose mode. */ diff --git a/Python/pytime.c b/Python/pytime.c index 02a9374..77db204 100644 --- a/Python/pytime.c +++ b/Python/pytime.c @@ -531,12 +531,8 @@ _PyTime_GetSystemClockWithInfo(_PyTime_t *t, _Py_clock_info_t *info) static int -pymonotonic_new(_PyTime_t *tp, _Py_clock_info_t *info, int raise) +pymonotonic(_PyTime_t *tp, _Py_clock_info_t *info, int raise) { -#ifdef Py_DEBUG - static int last_set = 0; - static _PyTime_t last = 0; -#endif #if defined(MS_WINDOWS) ULONGLONG result; @@ -628,12 +624,6 @@ pymonotonic_new(_PyTime_t *tp, _Py_clock_info_t *info, int raise) if (_PyTime_FromTimespec(tp, &ts, raise) < 0) return -1; #endif -#ifdef Py_DEBUG - /* monotonic clock cannot go backward */ - assert(!last_set || last <= *tp); - last = *tp; - last_set = 1; -#endif return 0; } @@ -641,7 +631,7 @@ _PyTime_t _PyTime_GetMonotonicClock(void) { _PyTime_t t; - if (pymonotonic_new(&t, NULL, 0) < 0) { + if (pymonotonic(&t, NULL, 0) < 0) { /* should not happen, _PyTime_Init() checked that monotonic clock at startup */ assert(0); @@ -655,7 +645,7 @@ _PyTime_GetMonotonicClock(void) int _PyTime_GetMonotonicClockWithInfo(_PyTime_t *tp, _Py_clock_info_t *info) { - return pymonotonic_new(tp, info, 1); + return pymonotonic(tp, info, 1); } int diff --git a/Tools/buildbot/test.bat b/Tools/buildbot/test.bat index 154dfa5..d1f5f52 100644 --- a/Tools/buildbot/test.bat +++ b/Tools/buildbot/test.bat @@ -1,15 +1,19 @@ -@rem Used by the buildbot "test" step.
-@setlocal
+@echo off
+rem Used by the buildbot "test" step.
+setlocal
-@set here=%~dp0
-@set rt_opts=-q -d
+set here=%~dp0
+set rt_opts=-q -d
+set regrtest_args=
:CheckOpts
-@if '%1'=='-x64' (set rt_opts=%rt_opts% %1) & shift & goto CheckOpts
-@if '%1'=='-d' (set rt_opts=%rt_opts% %1) & shift & goto CheckOpts
-@if '%1'=='-O' (set rt_opts=%rt_opts% %1) & shift & goto CheckOpts
-@if '%1'=='-q' (set rt_opts=%rt_opts% %1) & shift & goto CheckOpts
-@if '%1'=='+d' (set rt_opts=%rt_opts:-d=%) & shift & goto CheckOpts
-@if '%1'=='+q' (set rt_opts=%rt_opts:-q=%) & shift & goto CheckOpts
+if "%1"=="-x64" (set rt_opts=%rt_opts% %1) & shift & goto CheckOpts
+if "%1"=="-d" (set rt_opts=%rt_opts% %1) & shift & goto CheckOpts
+if "%1"=="-O" (set rt_opts=%rt_opts% %1) & shift & goto CheckOpts
+if "%1"=="-q" (set rt_opts=%rt_opts% %1) & shift & goto CheckOpts
+if "%1"=="+d" (set rt_opts=%rt_opts:-d=%) & shift & goto CheckOpts
+if "%1"=="+q" (set rt_opts=%rt_opts:-q=%) & shift & goto CheckOpts
+if NOT "%1"=="" (set regrtest_args=%regrtest_args% %1) & shift & goto CheckOpts
-call "%here%..\..\PCbuild\rt.bat" %rt_opts% -uall -rwW -n --timeout=3600 %1 %2 %3 %4 %5 %6 %7 %8 %9
+echo on
+call "%here%..\..\PCbuild\rt.bat" %rt_opts% -uall -rwW -n --timeout=3600 %regrtest_args%
diff --git a/Tools/msi/build.bat b/Tools/msi/build.bat index 008c937..5f53a1b 100644 --- a/Tools/msi/build.bat +++ b/Tools/msi/build.bat @@ -7,6 +7,7 @@ set BUILDX86= set BUILDX64=
set BUILDDOC=
set BUILDPX=
+set BUILDPACK=
:CheckOpts
if "%~1" EQU "-h" goto Help
@@ -14,6 +15,7 @@ if "%~1" EQU "-x86" (set BUILDX86=1) && shift && goto CheckOpts if "%~1" EQU "-x64" (set BUILDX64=1) && shift && goto CheckOpts
if "%~1" EQU "--doc" (set BUILDDOC=1) && shift && goto CheckOpts
if "%~1" EQU "--test-marker" (set BUILDPX=1) && shift && goto CheckOpts
+if "%~1" EQU "--pack" (set BUILDPACK=1) && shift && goto CheckOpts
if not defined BUILDX86 if not defined BUILDX64 (set BUILDX86=1) && (set BUILDX64=1)
@@ -41,6 +43,9 @@ set BUILD_CMD="%D%bundle\snapshot.wixproj" if defined BUILDPX (
set BUILD_CMD=%BUILD_CMD% /p:UseTestMarker=true
)
+if defined BUILDPACK (
+ set BUILD_CMD=%BUILD_CMD% /p:Pack=true
+)
if defined BUILDX86 (
"%PCBUILD%win32\python.exe" "%D%get_wix.py"
@@ -56,9 +61,10 @@ if defined BUILDX64 ( exit /B 0
:Help
-echo build.bat [-x86] [-x64] [--doc] [-h] [--test-marker]
+echo build.bat [-x86] [-x64] [--doc] [-h] [--test-marker] [--pack]
echo.
echo -x86 Build x86 installers
echo -x64 Build x64 installers
echo --doc Build CHM documentation
echo --test-marker Build installers with 'x' markers
+echo --pack Embed core MSIs into installer
diff --git a/Tools/msi/bundle/snapshot.wixproj b/Tools/msi/bundle/snapshot.wixproj index 8fe95a8..cc45043 100644 --- a/Tools/msi/bundle/snapshot.wixproj +++ b/Tools/msi/bundle/snapshot.wixproj @@ -9,9 +9,14 @@ <Import Project="..\msi.props" /> <PropertyGroup> + <DefineConstants Condition="'$(Pack)' != 'true'"> + $(DefineConstants);CompressMSI=no; + </DefineConstants> + <DefineConstants Condition="'$(Pack)' == 'true'"> + $(DefineConstants);CompressMSI=yes; + </DefineConstants> <DefineConstants> $(DefineConstants); - CompressMSI=no; CompressPDB=no; CompressMSI_D=no; </DefineConstants> diff --git a/Tools/msi/core/core_d.wxs b/Tools/msi/core/core_d.wxs index bb35f99..07e0397 100644 --- a/Tools/msi/core/core_d.wxs +++ b/Tools/msi/core/core_d.wxs @@ -1,6 +1,6 @@ <?xml version="1.0" encoding="UTF-8"?> <Wix xmlns="http://schemas.microsoft.com/wix/2006/wi"> - <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> + <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title_d)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" /> <MediaTemplate EmbedCab="yes" CompressionLevel="high" /> diff --git a/Tools/msi/core/core_pdb.wxs b/Tools/msi/core/core_pdb.wxs index ba72d95..c2c3178 100644 --- a/Tools/msi/core/core_pdb.wxs +++ b/Tools/msi/core/core_pdb.wxs @@ -1,6 +1,6 @@ <?xml version="1.0" encoding="UTF-8"?> <Wix xmlns="http://schemas.microsoft.com/wix/2006/wi"> - <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> + <Product Id="*" Language="!(loc.LCID)" Name="!(loc.TitlePdb)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" /> <MediaTemplate EmbedCab="yes" CompressionLevel="high" /> diff --git a/Tools/msi/dev/dev_d.wxs b/Tools/msi/dev/dev_d.wxs index c3cb2ea..c467aac 100644 --- a/Tools/msi/dev/dev_d.wxs +++ b/Tools/msi/dev/dev_d.wxs @@ -1,6 +1,6 @@ <?xml version="1.0" encoding="UTF-8"?> <Wix xmlns="http://schemas.microsoft.com/wix/2006/wi"> - <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> + <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title_d)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" /> <MediaTemplate EmbedCab="yes" CompressionLevel="high" /> diff --git a/Tools/msi/exe/exe_d.wxs b/Tools/msi/exe/exe_d.wxs index abcb012..eedb6bb 100644 --- a/Tools/msi/exe/exe_d.wxs +++ b/Tools/msi/exe/exe_d.wxs @@ -1,6 +1,6 @@ <?xml version="1.0" encoding="UTF-8"?> <Wix xmlns="http://schemas.microsoft.com/wix/2006/wi"> - <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> + <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title_d)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" /> <MediaTemplate EmbedCab="yes" CompressionLevel="high" /> diff --git a/Tools/msi/exe/exe_pdb.wxs b/Tools/msi/exe/exe_pdb.wxs index 5129ec0..f25094f 100644 --- a/Tools/msi/exe/exe_pdb.wxs +++ b/Tools/msi/exe/exe_pdb.wxs @@ -1,6 +1,6 @@ <?xml version="1.0" encoding="UTF-8"?> <Wix xmlns="http://schemas.microsoft.com/wix/2006/wi"> - <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> + <Product Id="*" Language="!(loc.LCID)" Name="!(loc.TitlePdb)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" /> <MediaTemplate EmbedCab="yes" CompressionLevel="high" /> diff --git a/Tools/msi/lib/lib_d.wxs b/Tools/msi/lib/lib_d.wxs index 5a5cf70..8a8a530 100644 --- a/Tools/msi/lib/lib_d.wxs +++ b/Tools/msi/lib/lib_d.wxs @@ -1,6 +1,6 @@ <?xml version="1.0" encoding="UTF-8"?> <Wix xmlns="http://schemas.microsoft.com/wix/2006/wi"> - <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> + <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title_d)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" /> <MediaTemplate EmbedCab="yes" CompressionLevel="high" /> diff --git a/Tools/msi/lib/lib_pdb.wxs b/Tools/msi/lib/lib_pdb.wxs index a2be0c9..8839e8a 100644 --- a/Tools/msi/lib/lib_pdb.wxs +++ b/Tools/msi/lib/lib_pdb.wxs @@ -1,6 +1,6 @@ <?xml version="1.0" encoding="UTF-8"?> <Wix xmlns="http://schemas.microsoft.com/wix/2006/wi"> - <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> + <Product Id="*" Language="!(loc.LCID)" Name="!(loc.TitlePdb)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" /> <MediaTemplate EmbedCab="yes" CompressionLevel="high" /> diff --git a/Tools/msi/make_zip.py b/Tools/msi/make_zip.py index 09ee491..c256008 100644 --- a/Tools/msi/make_zip.py +++ b/Tools/msi/make_zip.py @@ -3,6 +3,7 @@ import py_compile import re import sys import shutil +import stat import os import tempfile @@ -101,11 +102,16 @@ def copy_to_layout(target, rel_sources): else: for s, rel in rel_sources: + dest = target / rel try: - (target / rel).parent.mkdir(parents=True) + dest.parent.mkdir(parents=True) except FileExistsError: pass - shutil.copy(str(s), str(target / rel)) + if dest.is_file(): + dest.chmod(stat.S_IWRITE) + shutil.copy(str(s), str(dest)) + if dest.is_file(): + dest.chmod(stat.S_IWRITE) count += 1 return count diff --git a/Tools/msi/tcltk/tcltk_d.wxs b/Tools/msi/tcltk/tcltk_d.wxs index 7f5048f..01d0d24 100644 --- a/Tools/msi/tcltk/tcltk_d.wxs +++ b/Tools/msi/tcltk/tcltk_d.wxs @@ -1,6 +1,6 @@ <?xml version="1.0" encoding="UTF-8"?> <Wix xmlns="http://schemas.microsoft.com/wix/2006/wi"> - <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> + <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title_d)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" /> <MediaTemplate EmbedCab="yes" CompressionLevel="high" /> diff --git a/Tools/msi/tcltk/tcltk_pdb.wxs b/Tools/msi/tcltk/tcltk_pdb.wxs index 75c62bb..04454f3 100644 --- a/Tools/msi/tcltk/tcltk_pdb.wxs +++ b/Tools/msi/tcltk/tcltk_pdb.wxs @@ -1,6 +1,6 @@ <?xml version="1.0" encoding="UTF-8"?> <Wix xmlns="http://schemas.microsoft.com/wix/2006/wi"> - <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> + <Product Id="*" Language="!(loc.LCID)" Name="!(loc.TitlePdb)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" /> <MediaTemplate EmbedCab="yes" CompressionLevel="high" /> diff --git a/Tools/msi/test/test_d.wxs b/Tools/msi/test/test_d.wxs index a25afdd..a954876 100644 --- a/Tools/msi/test/test_d.wxs +++ b/Tools/msi/test/test_d.wxs @@ -1,6 +1,6 @@ <?xml version="1.0" encoding="UTF-8"?> <Wix xmlns="http://schemas.microsoft.com/wix/2006/wi"> - <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> + <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title_d)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" /> <MediaTemplate EmbedCab="yes" CompressionLevel="high" /> diff --git a/Tools/msi/test/test_pdb.wxs b/Tools/msi/test/test_pdb.wxs index 1510a6f..de634a3 100644 --- a/Tools/msi/test/test_pdb.wxs +++ b/Tools/msi/test/test_pdb.wxs @@ -1,6 +1,6 @@ <?xml version="1.0" encoding="UTF-8"?> <Wix xmlns="http://schemas.microsoft.com/wix/2006/wi"> - <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> + <Product Id="*" Language="!(loc.LCID)" Name="!(loc.TitlePdb)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)"> <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" /> <MediaTemplate EmbedCab="yes" CompressionLevel="high" /> |