diff options
-rw-r--r-- | Doc/c-api/init_config.rst | 2 | ||||
-rw-r--r-- | Doc/library/asyncio-dev.rst | 2 | ||||
-rw-r--r-- | Doc/library/asyncio-eventloop.rst | 2 | ||||
-rw-r--r-- | Doc/library/development.rst | 4 | ||||
-rw-r--r-- | Doc/library/devmode.rst | 214 | ||||
-rw-r--r-- | Doc/library/exceptions.rst | 16 | ||||
-rw-r--r-- | Doc/library/faulthandler.rst | 3 | ||||
-rw-r--r-- | Doc/library/stdtypes.rst | 8 | ||||
-rw-r--r-- | Doc/library/sys.rst | 15 | ||||
-rw-r--r-- | Doc/using/cmdline.rst | 26 | ||||
-rw-r--r-- | Doc/whatsnew/3.7.rst | 10 | ||||
-rw-r--r-- | Doc/whatsnew/3.9.rst | 7 | ||||
-rw-r--r-- | Include/cpython/initconfig.h | 8 |
13 files changed, 268 insertions, 49 deletions
diff --git a/Doc/c-api/init_config.rst b/Doc/c-api/init_config.rst index 79a8815..108bd2c 100644 --- a/Doc/c-api/init_config.rst +++ b/Doc/c-api/init_config.rst @@ -466,7 +466,7 @@ PyConfig .. c:member:: int dev_mode - Development mode: see :option:`-X dev <-X>`. + If non-zero, enable the :ref:`Python Development Mode <devmode>`. .. c:member:: int dump_refs diff --git a/Doc/library/asyncio-dev.rst b/Doc/library/asyncio-dev.rst index 101e781..ff51c4f 100644 --- a/Doc/library/asyncio-dev.rst +++ b/Doc/library/asyncio-dev.rst @@ -25,7 +25,7 @@ There are several ways to enable asyncio debug mode: * Setting the :envvar:`PYTHONASYNCIODEBUG` environment variable to ``1``. -* Using the :option:`-X` ``dev`` Python command line option. +* Using the :ref:`Python Development Mode <devmode>`. * Passing ``debug=True`` to :func:`asyncio.run`. diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst index 25a3692..0029d94 100644 --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -1200,7 +1200,7 @@ Enabling debug mode .. versionchanged:: 3.7 - The new ``-X dev`` command line option can now also be used + The new :ref:`Python Development Mode <devmode>` can now also be used to enable the debug mode. .. seealso:: diff --git a/Doc/library/development.rst b/Doc/library/development.rst index ab34e1f..9edce75 100644 --- a/Doc/library/development.rst +++ b/Doc/library/development.rst @@ -18,12 +18,10 @@ The list of modules described in this chapter is: typing.rst pydoc.rst + devmode.rst doctest.rst unittest.rst unittest.mock.rst unittest.mock-examples.rst 2to3.rst test.rst - -See also the Python development mode: the :option:`-X` ``dev`` option and -:envvar:`PYTHONDEVMODE` environment variable. diff --git a/Doc/library/devmode.rst b/Doc/library/devmode.rst new file mode 100644 index 0000000..d5a40cd --- /dev/null +++ b/Doc/library/devmode.rst @@ -0,0 +1,214 @@ +.. _devmode: + +Python Development Mode +======================= + +.. versionadded:: 3.7 + +The Python Development Mode introduces additional runtime checks that are too +expensive to be enabled by default. It should not be more verbose than the +default if the code is correct; new warnings are only emitted when an issue is +detected. + +It can be enabled using the :option:`-X dev <-X>` command line option or by +setting the :envvar:`PYTHONDEVMODE` environment variable to ``1``. + +Effects of the Python Development Mode +====================================== + +Enabling the Python Development Mode is similar to the following command, but +with additional effects described below:: + + PYTHONMALLOC=debug PYTHONASYNCIODEBUG=1 python3 -W default -X faulthandler + +Effects of the Python Development Mode: + +* Add ``default`` :ref:`warning filter <describing-warning-filters>`. The + following warnings are shown: + + * :exc:`DeprecationWarning` + * :exc:`ImportWarning` + * :exc:`PendingDeprecationWarning` + * :exc:`ResourceWarning` + + Normally, the above warnings are filtered by the default :ref:`warning + filters <describing-warning-filters>`. + + It behaves as if the :option:`-W default <-W>` command line option is used. + + Use the :option:`-W error <-W>` command line option or set the + :envvar:`PYTHONWARNINGS` environment variable to ``error`` to treat warnings + as errors. + +* Install debug hooks on memory allocators to check for: + + * Buffer underflow + * Buffer overflow + * Memory allocator API violation + * Unsafe usage of the GIL + + See the :c:func:`PyMem_SetupDebugHooks` C function. + + It behaves as if the :envvar:`PYTHONMALLOC` environment variable is set to + ``debug``. + + To enable the Python Development Mode without installing debug hooks on + memory allocators, set the :envvar:`PYTHONMALLOC` environment variable to + ``default``. + +* Call :func:`faulthandler.enable` at Python startup to install handlers for + the :const:`SIGSEGV`, :const:`SIGFPE`, :const:`SIGABRT`, :const:`SIGBUS` and + :const:`SIGILL` signals to dump the Python traceback on a crash. + + It behaves as if the :option:`-X faulthandler <-X>` command line option is + used or if the :envvar:`PYTHONFAULTHANDLER` environment variable is set to + ``1``. + +* Enable :ref:`asyncio debug mode <asyncio-debug-mode>`. For example, + :mod:`asyncio` checks for coroutines that were not awaited and logs them. + + It behaves as if the :envvar:`PYTHONASYNCIODEBUG` environment variable is set + to ``1``. + +* Check the *encoding* and *errors* arguments for string encoding and decoding + operations. Examples: :func:`open`, :meth:`str.encode` and + :meth:`bytes.decode`. + + By default, for best performance, the *errors* argument is only checked at + the first encoding/decoding error and the *encoding* argument is sometimes + ignored for empty strings. + +* The :class:`io.IOBase` destructor logs ``close()`` exceptions. +* Set the :attr:`~sys.flags.dev_mode` attribute of :attr:`sys.flags` to + ``True``. + +The Python Development Mode does not enable the :mod:`tracemalloc` module by +default, because the overhead cost (to performance and memory) would be too +large. Enabling the :mod:`tracemalloc` module provides additional information +on the origin of some errors. For example, :exc:`ResourceWarning` logs the +traceback where the resource was allocated, and a buffer overflow error logs +the traceback where the memory block was allocated. + +The Python Development Mode does not prevent the :option:`-O` command line +option from removing :keyword:`assert` statements nor from setting +:const:`__debug__` to ``False``. + +.. versionchanged:: 3.8 + The :class:`io.IOBase` destructor now logs ``close()`` exceptions. + +.. versionchanged:: 3.9 + The *encoding* and *errors* arguments are now checked for string encoding + and decoding operations. + + +ResourceWarning Example +======================= + +Example of a script counting the number of lines of the text file specified in +the command line:: + + import sys + + def main(): + fp = open(sys.argv[1]) + nlines = len(fp.readlines()) + print(nlines) + # The file is closed implicitly + + if __name__ == "__main__": + main() + +The script does not close the file explicitly. By default, Python does not emit +any warning. Example using README.txt, which has 269 lines: + +.. code-block:: shell-session + + $ python3 script.py README.txt + 269 + +Enabling the Python Development Mode displays a :exc:`ResourceWarning` warning: + +.. code-block:: shell-session + + $ python3 -X dev script.py README.txt + 269 + script.py:10: ResourceWarning: unclosed file <_io.TextIOWrapper name='README.rst' mode='r' encoding='UTF-8'> + main() + ResourceWarning: Enable tracemalloc to get the object allocation traceback + +In addition, enabling :mod:`tracemalloc` shows the line where the file was +opened: + +.. code-block:: shell-session + + $ python3 -X dev -X tracemalloc=5 script.py README.rst + 269 + script.py:10: ResourceWarning: unclosed file <_io.TextIOWrapper name='README.rst' mode='r' encoding='UTF-8'> + main() + Object allocated at (most recent call last): + File "script.py", lineno 10 + main() + File "script.py", lineno 4 + fp = open(sys.argv[1]) + +The fix is to close explicitly the file. Example using a context manager:: + + def main(): + # Close the file explicitly when exiting the with block + with open(sys.argv[1]) as fp: + nlines = len(fp.readlines()) + print(nlines) + +Not closing a resource explicitly can leave a resource open for way longer than +expected; it can cause severe issues upon exiting Python. It is bad in +CPython, but it is even worse in PyPy. Closing resources explicitly makes an +application more deterministic and more reliable. + + +Bad file descriptor error example +================================= + +Script displaying the first line of itself:: + + import os + + def main(): + fp = open(__file__) + firstline = fp.readline() + print(firstline.rstrip()) + os.close(fp.fileno()) + # The file is closed implicitly + + main() + +By default, Python does not emit any warning: + +.. code-block:: shell-session + + $ python3 script.py + import os + +The Python Development Mode shows a :exc:`ResourceWarning` and logs a "Bad file +descriptor" error when finalizing the file object: + +.. code-block:: shell-session + + $ python3 script.py + import os + script.py:10: ResourceWarning: unclosed file <_io.TextIOWrapper name='script.py' mode='r' encoding='UTF-8'> + main() + ResourceWarning: Enable tracemalloc to get the object allocation traceback + Exception ignored in: <_io.TextIOWrapper name='script.py' mode='r' encoding='UTF-8'> + Traceback (most recent call last): + File "script.py", line 10, in <module> + main() + OSError: [Errno 9] Bad file descriptor + +``os.close(fp.fileno())`` closes the file descriptor. When the file object +finalizer tries to close the file descriptor again, it fails with the ``Bad +file descriptor`` error. A file descriptor must be closed only once. In the +worst case scenario, closing it twice can lead to a crash (see :issue:`18748` +for an example). + +The fix is to remove the ``os.close(fp.fileno())`` line, or open the file with +``closefd=False``. diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst index 52a505e..df2cda9 100644 --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -688,6 +688,10 @@ The following exceptions are used as warning categories; see the Base class for warnings about deprecated features when those warnings are intended for other Python developers. + Ignored by the default warning filters, except in the ``__main__`` module + (:pep:`565`). Enabling the :ref:`Python Development Mode <devmode>` shows + this warning. + .. exception:: PendingDeprecationWarning @@ -699,6 +703,9 @@ The following exceptions are used as warning categories; see the upcoming deprecation is unusual, and :exc:`DeprecationWarning` is preferred for already active deprecations. + Ignored by the default warning filters. Enabling the :ref:`Python + Development Mode <devmode>` shows this warning. + .. exception:: SyntaxWarning @@ -720,6 +727,9 @@ The following exceptions are used as warning categories; see the Base class for warnings about probable mistakes in module imports. + Ignored by the default warning filters. Enabling the :ref:`Python + Development Mode <devmode>` shows this warning. + .. exception:: UnicodeWarning @@ -733,8 +743,10 @@ The following exceptions are used as warning categories; see the .. exception:: ResourceWarning - Base class for warnings related to resource usage. Ignored by the default - warning filters. + Base class for warnings related to resource usage. + + Ignored by the default warning filters. Enabling the :ref:`Python + Development Mode <devmode>` shows this warning. .. versionadded:: 3.2 diff --git a/Doc/library/faulthandler.rst b/Doc/library/faulthandler.rst index b588dfa..59274c1 100644 --- a/Doc/library/faulthandler.rst +++ b/Doc/library/faulthandler.rst @@ -40,6 +40,9 @@ alternatively be passed to :func:`faulthandler.enable`. The module is implemented in C, so tracebacks can be dumped on a crash or when Python is deadlocked. +The :ref:`Python Development Mode <devmode>` calls :func:`faulthandler.enable` +at Python startup. + Dumping the traceback --------------------- diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index 3e25faa..fd3401f 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -1559,8 +1559,8 @@ expression support in the :mod:`re` module). list of possible encodings, see section :ref:`standard-encodings`. By default, the *errors* argument is not checked for best performances, but - only used at the first encoding error. Enable the development mode - (:option:`-X` ``dev`` option), or use a debug build, to check *errors*. + only used at the first encoding error. Enable the :ref:`Python Development + Mode <devmode>`, or use a debug build to check *errors*. .. versionchanged:: 3.1 Support for keyword arguments added. @@ -2596,8 +2596,8 @@ arbitrary binary data. list of possible encodings, see section :ref:`standard-encodings`. By default, the *errors* argument is not checked for best performances, but - only used at the first decoding error. Enable the development mode - (:option:`-X` ``dev`` option), or use a debug build, to check *errors*. + only used at the first decoding error. Enable the :ref:`Python Development + Mode <devmode>`, or use a debug build to check *errors*. .. note:: diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst index 351a8e4..d28b356 100644 --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -428,9 +428,9 @@ always available. The :term:`named tuple` *flags* exposes the status of command line flags. The attributes are read only. - ============================= ============================= + ============================= ================================================================ attribute flag - ============================= ============================= + ============================= ================================================================ :const:`debug` :option:`-d` :const:`inspect` :option:`-i` :const:`interactive` :option:`-i` @@ -444,9 +444,9 @@ always available. :const:`bytes_warning` :option:`-b` :const:`quiet` :option:`-q` :const:`hash_randomization` :option:`-R` - :const:`dev_mode` :option:`-X` ``dev`` - :const:`utf8_mode` :option:`-X` ``utf8`` - ============================= ============================= + :const:`dev_mode` :option:`-X dev <-X>` (:ref:`Python Development Mode <devmode>`) + :const:`utf8_mode` :option:`-X utf8 <-X>` + ============================= ================================================================ .. versionchanged:: 3.2 Added ``quiet`` attribute for the new :option:`-q` flag. @@ -461,8 +461,9 @@ always available. Added ``isolated`` attribute for :option:`-I` ``isolated`` flag. .. versionchanged:: 3.7 - Added ``dev_mode`` attribute for the new :option:`-X` ``dev`` flag - and ``utf8_mode`` attribute for the new :option:`-X` ``utf8`` flag. + Added the ``dev_mode`` attribute for the new :ref:`Python Development + Mode <devmode>` and the ``utf8_mode`` attribute for the new :option:`-X` + ``utf8`` flag. .. data:: float_info diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index 9e14980..146003b 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -442,24 +442,9 @@ Miscellaneous options nested imports). Note that its output may be broken in multi-threaded application. Typical usage is ``python3 -X importtime -c 'import asyncio'``. See also :envvar:`PYTHONPROFILEIMPORTTIME`. - * ``-X dev``: enable CPython's "development mode", introducing additional - runtime checks which are too expensive to be enabled by default. It should - not be more verbose than the default if the code is correct: new warnings - are only emitted when an issue is detected. Effect of the developer mode: - - * Check *encoding* and *errors* arguments on string encoding and decoding - operations. Examples: :func:`open`, :meth:`str.encode` and - :meth:`bytes.decode`. - * Add ``default`` warning filter, as :option:`-W` ``default``. - * Install debug hooks on memory allocators: see the - :c:func:`PyMem_SetupDebugHooks` C function. - * Enable the :mod:`faulthandler` module to dump the Python traceback - on a crash. - * Enable :ref:`asyncio debug mode <asyncio-debug-mode>`. - * Set the :attr:`~sys.flags.dev_mode` attribute of :attr:`sys.flags` to - ``True``. - * :class:`io.IOBase` destructor logs ``close()`` exceptions. - + * ``-X dev``: enable :ref:`Python Development Mode <devmode>`, introducing + additional runtime checks that are too expensive to be enabled by + default. * ``-X utf8`` enables UTF-8 mode for operating system interfaces, overriding the default locale-aware mode. ``-X utf8=0`` explicitly disables UTF-8 mode (even when it would otherwise activate automatically). @@ -890,8 +875,9 @@ conflict. .. envvar:: PYTHONDEVMODE - If this environment variable is set to a non-empty string, enable the - CPython "development mode". See the :option:`-X` ``dev`` option. + If this environment variable is set to a non-empty string, enable + :ref:`Python Development Mode <devmode>`, introducing additional runtime + checks that are too expensive to be enabled by default. .. versionadded:: 3.7 diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst index 8a70fe2..04cfa57 100644 --- a/Doc/whatsnew/3.7.rst +++ b/Doc/whatsnew/3.7.rst @@ -102,7 +102,7 @@ CPython implementation improvements: * :ref:`PEP 538 <whatsnew37-pep538>`, legacy C locale coercion * :ref:`PEP 540 <whatsnew37-pep540>`, forced UTF-8 runtime mode * :ref:`PEP 552 <whatsnew37-pep552>`, deterministic .pycs -* :ref:`the new development runtime mode <whatsnew37-devmode>` +* :ref:`New Python Development Mode <whatsnew37-devmode>` * :ref:`PEP 565 <whatsnew37-pep565>`, improved :exc:`DeprecationWarning` handling @@ -479,15 +479,15 @@ Three new translations have been added: .. _whatsnew37-devmode: -Development Runtime Mode: -X dev +Python Development Mode (-X dev) -------------------------------- The new :option:`-X` ``dev`` command line option or the new :envvar:`PYTHONDEVMODE` environment variable can be used to enable -CPython's *development mode*. When in development mode, CPython performs +:ref:`Python Development Mode <devmode>`. When in development mode, Python performs additional runtime checks that are too expensive to be enabled by default. -See :option:`-X` ``dev`` documentation for the full description of the effects -of this mode. +See :ref:`Python Development Mode <devmode>` documentation for the full +description. Other Language Changes diff --git a/Doc/whatsnew/3.9.rst b/Doc/whatsnew/3.9.rst index a6e938f..ff5cb14 100644 --- a/Doc/whatsnew/3.9.rst +++ b/Doc/whatsnew/3.9.rst @@ -90,9 +90,10 @@ Other Language Changes in this case. (Contributed by Victor Stinner in :issue:`20443`.) -* In development mode and in debug build, *encoding* and *errors* arguments are - now checked on string encoding and decoding operations. Examples: - :func:`open`, :meth:`str.encode` and :meth:`bytes.decode`. +* In the :ref:`Python Development Mode <devmode>` and in debug build, the + *encoding* and *errors* arguments are now checked for string encoding and + decoding operations. Examples: :func:`open`, :meth:`str.encode` and + :meth:`bytes.decode`. By default, for best performance, the *errors* argument is only checked at the first encoding/decoding error and the *encoding* argument is sometimes diff --git a/Include/cpython/initconfig.h b/Include/cpython/initconfig.h index 4b5ceaf..54e6623 100644 --- a/Include/cpython/initconfig.h +++ b/Include/cpython/initconfig.h @@ -113,7 +113,11 @@ typedef struct { "POSIX", otherwise it is set to 0. Inherit Py_UTF8Mode value value. */ int utf8_mode; - int dev_mode; /* Development mode. PYTHONDEVMODE, -X dev */ + /* If non-zero, enable the Python Development Mode. + + Set to 1 by the -X dev command line option. Set by the PYTHONDEVMODE + environment variable. */ + int dev_mode; /* Memory allocator: PYTHONMALLOC env var. See PyMemAllocatorName for valid values. */ @@ -131,7 +135,7 @@ typedef struct { int isolated; /* Isolated mode? see PyPreConfig.isolated */ int use_environment; /* Use environment variables? see PyPreConfig.use_environment */ - int dev_mode; /* Development mode? See PyPreConfig.dev_mode */ + int dev_mode; /* Python Development Mode? See PyPreConfig.dev_mode */ /* Install signal handlers? Yes by default. */ int install_signal_handlers; |