1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
|
.. _devmode:
Python Development Mode
=======================
.. versionadded:: 3.7
The Python Development Mode introduces additional runtime checks that are too
expensive to be enabled by default. It should not be more verbose than the
default if the code is correct; new warnings are only emitted when an issue is
detected.
It can be enabled using the :option:`-X dev <-X>` command line option or by
setting the :envvar:`PYTHONDEVMODE` environment variable to ``1``.
See also :ref:`Python debug build <debug-build>`.
Effects of the Python Development Mode
--------------------------------------
Enabling the Python Development Mode is similar to the following command, but
with additional effects described below::
PYTHONMALLOC=debug PYTHONASYNCIODEBUG=1 python3 -W default -X faulthandler
Effects of the Python Development Mode:
* Add ``default`` :ref:`warning filter <describing-warning-filters>`. The
following warnings are shown:
* :exc:`DeprecationWarning`
* :exc:`ImportWarning`
* :exc:`PendingDeprecationWarning`
* :exc:`ResourceWarning`
Normally, the above warnings are filtered by the default :ref:`warning
filters <describing-warning-filters>`.
It behaves as if the :option:`-W default <-W>` command line option is used.
Use the :option:`-W error <-W>` command line option or set the
:envvar:`PYTHONWARNINGS` environment variable to ``error`` to treat warnings
as errors.
* Install debug hooks on memory allocators to check for:
* Buffer underflow
* Buffer overflow
* Memory allocator API violation
* Unsafe usage of the GIL
See the :c:func:`PyMem_SetupDebugHooks` C function.
It behaves as if the :envvar:`PYTHONMALLOC` environment variable is set to
``debug``.
To enable the Python Development Mode without installing debug hooks on
memory allocators, set the :envvar:`PYTHONMALLOC` environment variable to
``default``.
* Call :func:`faulthandler.enable` at Python startup to install handlers for
the :const:`~signal.SIGSEGV`, :const:`~signal.SIGFPE`,
:const:`~signal.SIGABRT`, :const:`~signal.SIGBUS` and
:const:`~signal.SIGILL` signals to dump the Python traceback on a crash.
It behaves as if the :option:`-X faulthandler <-X>` command line option is
used or if the :envvar:`PYTHONFAULTHANDLER` environment variable is set to
``1``.
* Enable :ref:`asyncio debug mode <asyncio-debug-mode>`. For example,
:mod:`asyncio` checks for coroutines that were not awaited and logs them.
It behaves as if the :envvar:`PYTHONASYNCIODEBUG` environment variable is set
to ``1``.
* Check the *encoding* and *errors* arguments for string encoding and decoding
operations. Examples: :func:`open`, :meth:`str.encode` and
:meth:`bytes.decode`.
By default, for best performance, the *errors* argument is only checked at
the first encoding/decoding error and the *encoding* argument is sometimes
ignored for empty strings.
* The :class:`io.IOBase` destructor logs ``close()`` exceptions.
* Set the :attr:`~sys.flags.dev_mode` attribute of :data:`sys.flags` to
``True``.
The Python Development Mode does not enable the :mod:`tracemalloc` module by
default, because the overhead cost (to performance and memory) would be too
large. Enabling the :mod:`tracemalloc` module provides additional information
on the origin of some errors. For example, :exc:`ResourceWarning` logs the
traceback where the resource was allocated, and a buffer overflow error logs
the traceback where the memory block was allocated.
The Python Development Mode does not prevent the :option:`-O` command line
option from removing :keyword:`assert` statements nor from setting
:const:`__debug__` to ``False``.
The Python Development Mode can only be enabled at the Python startup. Its
value can be read from :data:`sys.flags.dev_mode <sys.flags>`.
.. versionchanged:: 3.8
The :class:`io.IOBase` destructor now logs ``close()`` exceptions.
.. versionchanged:: 3.9
The *encoding* and *errors* arguments are now checked for string encoding
and decoding operations.
ResourceWarning Example
-----------------------
Example of a script counting the number of lines of the text file specified in
the command line::
import sys
def main():
fp = open(sys.argv[1])
nlines = len(fp.readlines())
print(nlines)
# The file is closed implicitly
if __name__ == "__main__":
main()
The script does not close the file explicitly. By default, Python does not emit
any warning. Example using README.txt, which has 269 lines:
.. code-block:: shell-session
$ python3 script.py README.txt
269
Enabling the Python Development Mode displays a :exc:`ResourceWarning` warning:
.. code-block:: shell-session
$ python3 -X dev script.py README.txt
269
script.py:10: ResourceWarning: unclosed file <_io.TextIOWrapper name='README.rst' mode='r' encoding='UTF-8'>
main()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
In addition, enabling :mod:`tracemalloc` shows the line where the file was
opened:
.. code-block:: shell-session
$ python3 -X dev -X tracemalloc=5 script.py README.rst
269
script.py:10: ResourceWarning: unclosed file <_io.TextIOWrapper name='README.rst' mode='r' encoding='UTF-8'>
main()
Object allocated at (most recent call last):
File "script.py", lineno 10
main()
File "script.py", lineno 4
fp = open(sys.argv[1])
The fix is to close explicitly the file. Example using a context manager::
def main():
# Close the file explicitly when exiting the with block
with open(sys.argv[1]) as fp:
nlines = len(fp.readlines())
print(nlines)
Not closing a resource explicitly can leave a resource open for way longer than
expected; it can cause severe issues upon exiting Python. It is bad in
CPython, but it is even worse in PyPy. Closing resources explicitly makes an
application more deterministic and more reliable.
Bad file descriptor error example
---------------------------------
Script displaying the first line of itself::
import os
def main():
fp = open(__file__)
firstline = fp.readline()
print(firstline.rstrip())
os.close(fp.fileno())
# The file is closed implicitly
main()
By default, Python does not emit any warning:
.. code-block:: shell-session
$ python3 script.py
import os
The Python Development Mode shows a :exc:`ResourceWarning` and logs a "Bad file
descriptor" error when finalizing the file object:
.. code-block:: shell-session
$ python3 -X dev script.py
import os
script.py:10: ResourceWarning: unclosed file <_io.TextIOWrapper name='script.py' mode='r' encoding='UTF-8'>
main()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Exception ignored in: <_io.TextIOWrapper name='script.py' mode='r' encoding='UTF-8'>
Traceback (most recent call last):
File "script.py", line 10, in <module>
main()
OSError: [Errno 9] Bad file descriptor
``os.close(fp.fileno())`` closes the file descriptor. When the file object
finalizer tries to close the file descriptor again, it fails with the ``Bad
file descriptor`` error. A file descriptor must be closed only once. In the
worst case scenario, closing it twice can lead to a crash (see :issue:`18748`
for an example).
The fix is to remove the ``os.close(fp.fileno())`` line, or open the file with
``closefd=False``.
|
