diff options
| author | morotti <r.morotti@gmail.com> | 2024-10-04 23:51:22 (GMT) |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2024-10-04 23:51:22 (GMT) |
| commit | 6efd95c4650ec7c2fb5522b352c74a9d44370fe0 (patch) | |
| tree | 033dfeccf8965b37f0764e6e246fff0ff6aaed41 | |
| parent | 8bcf118dcb2cab88acc8a6dffb7968b1854802b4 (diff) | |
| download | cpython-6efd95c4650ec7c2fb5522b352c74a9d44370fe0.zip cpython-6efd95c4650ec7c2fb5522b352c74a9d44370fe0.tar.gz cpython-6efd95c4650ec7c2fb5522b352c74a9d44370fe0.tar.bz2 | |
gh-117151: increase default buffer size of shutil.copyfileobj() to 256k. (GH-119783)
* gh-117151: increase default buffer size of shutil.copyfileobj() to 256k.
it was set to 16k in the 1990s.
it was raised to 64k in 2019. the discussion at the time mentioned another 5% improvement by raising to 128k and settled for a very conservative setting.
it's 2024 now, I think it should be revisited to match modern hardware. I am measuring 0-15% performance improvement when raising to 256k on various types of disk. there is no downside as far as I can tell.
this function is only intended for sequential copy of full files (or file like objects). it's the typical use case that benefits from larger operations.
for reference, I came across this function while trying to profile pip that is using it to copy files when installing python packages.
* add news
---------
Co-authored-by: rmorotti <romain.morotti@man.com>
| -rw-r--r-- | Lib/shutil.py | 2 | ||||
| -rw-r--r-- | Misc/NEWS.d/next/Library/2024-10-03-05-00-25.gh-issue-117151.Prdw_W.rst | 3 |
2 files changed, 4 insertions, 1 deletions
diff --git a/Lib/shutil.py b/Lib/shutil.py index dab3ca5..dd3e0e0 100644 --- a/Lib/shutil.py +++ b/Lib/shutil.py @@ -44,7 +44,7 @@ if sys.platform == 'win32': else: _winapi = None -COPY_BUFSIZE = 1024 * 1024 if _WINDOWS else 64 * 1024 +COPY_BUFSIZE = 1024 * 1024 if _WINDOWS else 256 * 1024 # This should never be removed, see rationale in: # https://bugs.python.org/issue43743#msg393429 _USE_CP_SENDFILE = (hasattr(os, "sendfile") diff --git a/Misc/NEWS.d/next/Library/2024-10-03-05-00-25.gh-issue-117151.Prdw_W.rst b/Misc/NEWS.d/next/Library/2024-10-03-05-00-25.gh-issue-117151.Prdw_W.rst new file mode 100644 index 0000000..a7d6251 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-10-03-05-00-25.gh-issue-117151.Prdw_W.rst @@ -0,0 +1,3 @@ +The default buffer size used by :func:`shutil.copyfileobj` has been +increased from 64k to 256k on non-Windows platforms. It was already larger +on Windows. |