bpo-31530: stop crashes when iterating over a file on multiple threadsbenjamin-iteration-torture

Multiple threads iterating over a file can corrupt the file's internal readahead
buffer resulting in crashes. To fix this, cache buffer state thread-locally for
the duration of a file_iternext call and only update the file's internal state
after reading completes.

No attempt is made to define or provide "reasonable" semantics for iterating
over a file on multiple threads. (Non-crashing) races are still
present. Duplicated, corrupt, and missing data will happen.

1 files changed, 10 insertions, 0 deletions

diff --git a/Lib/test/test_file2k.py b/Lib/test/test_file2k.py
index e39ef70..3b59ed3 100644
--- a/Lib/test/test_file2k.py
+++ b/Lib/test/test_file2k.py
@@ -652,6 +652,16 @@ class FileThreadingTests(unittest.TestCase):
             self.f.writelines('')
         self._test_close_open_io(io_func)
 
+    def test_iteration_torture(self):
+        with open(self.filename, "wb") as fp:
+            for i in xrange(2**20):
+                fp.write(b"0"*50 + b"\n")
+        with open(self.filename, "rb") as f:
+            def it():
+                for l in f:
+                    pass
+            self._run_workers(it, 10)
+
 
 @unittest.skipUnless(os.name == 'posix', 'test requires a posix system.')
 class TestFileSignalEINTR(unittest.TestCase):