summaryrefslogtreecommitdiffstats
path: root/Lib/urllib
Commit message (Collapse)AuthorAgeFilesLines
* gh-67693: Fix urlunparse() and urlunsplit() for URIs with path starting with ↵Serhiy Storchaka2024-05-141-1/+1
| | | | multiple slashes and no authority (GH-113563)
* gh-99730: urllib.request: Keep HEAD method on redirect (GH-99731)Harmen Stoppels2024-05-011-0/+1
|
* gh-116764: Fix regressions in urllib.parse.parse_qsl() (GH-116801)Serhiy Storchaka2024-03-161-1/+5
| | | | | | | | * Restore support of None and other false values. * Raise TypeError for non-zero integers and non-empty sequences. The regressions were introduced in gh-74668 (bdba8ef42b15e651dc23374a08143cc2b4c4657d).
* gh-74668: Fix support of bytes in urllib.parse.parse_qsl() (GH-115771)Serhiy Storchaka2024-03-051-24/+26
| | | | urllib.parse functions parse_qs() and parse_qsl() now support bytes arguments containing raw and percent-encoded non-ASCII data.
* gh-115197: Stop resolving host in urllib.request proxy bypass (GH-115210)Weii Wang2024-02-281-42/+35
| | | Use of a proxy is intended to defer DNS for the hosts to the proxy itself, rather than a potential for information leak of the host doing DNS resolution itself for any reason. Proxy bypass lists are strictly name based. Most implementations of proxy support agree.
* gh-91539: Small performance improvement of ↵Raphaël Marinier2024-01-151-1/+1
| | | | | urrlib.request.getproxies_environment() (#108771) Small performance improvement of getproxies_environment() when there are many environment variables. In a benchmark with 5k environment variables not related to proxies, and 5 specifying proxies, we get a 10% walltime improvement.
* GH-104554: Add RTSPS support to `urllib/parse.py` (#104605)zentarim2023-06-131-5/+5
| | | | | | | | | | | | | * GH-104554: Add RTSPS support to `urllib/parse.py` RTSPS is the permanent scheme defined in https://www.iana.org/assignments/uri-schemes/uri-schemes.xhtml alongside RTSP and RTSPU schemes. * 📜🤖 Added by blurb_it. --------- Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
* gh-105382: Remove urllib.request cafile parameter (#105384)Victor Stinner2023-06-061-28/+2
| | | | Remove cafile, capath and cadefault parameters of the urllib.request.urlopen() function, deprecated in Python 3.6.
* gh-102153: Start stripping C0 control and space chars in `urlsplit` (#102508)Illia Volochii2023-05-171-0/+12
| | | | | | | | | `urllib.parse.urlsplit` has already been respecting the WHATWG spec a bit #25595. This adds more sanitizing to respect the "Remove any leading C0 control or space from input" [rule](https://url.spec.whatwg.org/#url-parsing:~:text=Remove%20any%20leading%20and%20trailing%20C0%20control%20or%20space%20from%20input.) in response to [CVE-2023-24329](https://nvd.nist.gov/vuln/detail/CVE-2023-24329). --------- Co-authored-by: Gregory P. Smith [Google] <greg@krypto.org>
* gh-103848: Adds checks to ensure that bracketed hosts found by urlsplit are ↵JohnJamesUtley2023-05-101-1/+15
| | | | | | | | | of IPv6 or IPvFuture format (#103849) * Adds checks to ensure that bracketed hosts found by urlsplit are of IPv6 or IPvFuture format --------- Co-authored-by: Gregory P. Smith <greg@krypto.org>
* gh-104139: Add itms-services to uses_netloc urllib.parse. (#104312)Gregory P. Smith2023-05-091-1/+1
| | | | Teach unsplit to retain the `"//"` when assembling `itms-services://?action=generate-bugs` style [Apple Platform Deployment](https://support.apple.com/en-gb/guide/deployment/depce7cefc4d/web) URLs.
* gh-81403: Fix for CacheFTPHandler in urllib (#13951)Dan Hemberger2023-04-231-0/+6
| | | | | | | | | | | | | | bpo-37222: Fix for CacheFTPHandler in urllib A call to FTP.ntransfercmd must be followed by FTP.voidresp to clear the "end transfer" message. Without this, the client and server get out of sync, which will result in an error if the FTP instance is reused to open a second URL. This scenario occurs for even the most basic usage of CacheFTPHandler. Reverts the patch merged as a resolution to bpo-16270 and adds a test case for the CacheFTPHandler in test_urllib2net.py. Co-authored-by: Senthil Kumaran <senthil@python.org>
* gh-99352: Respect `http.client.HTTPConnection.debuglevel` in ↵Wheeler Law2023-04-211-3/+4
| | | | | | | | | | | | | | | | | | | | | `urllib.request.AbstractHTTPHandler` (#99353) * bugfix: let the HTTP- and HTTPSHandlers respect the value of http.client.HTTPConnection.debuglevel * add tests * add news * ReSTify NEWS and reword a bit. * Address Review Comments. * Use mock.patch.object instead of settting the module level value. * Used test values to assert the debuglevel. --------- Co-authored-by: Gregory P. Smith <greg@krypto.org> Co-authored-by: Senthil Kumaran <senthil@python.org>
* gh-101936: Update the default value of fp from io.StringIO to io.BytesIO ↵Vo Hoang Long2023-02-211-1/+1
| | | | | (gh-102100) Co-authored-by: Long Vo <long.vo@linecorp.com>
* gh-88500: Reduce memory use of `urllib.unquote` (#96763)Gregory P. Smith2022-12-111-11/+19
| | | | | | | | | | | `urllib.unquote_to_bytes` and `urllib.unquote` could both potentially generate `O(len(string))` intermediate `bytes` or `str` objects while computing the unquoted final result depending on the input provided. As Python objects are relatively large, this could consume a lot of ram. This switches the implementation to using an expanding `bytearray` and a generator internally instead of precomputed `split()` style operations. Microbenchmarks with some antagonistic inputs like `mess = "\u0141%%%20a%fe"*1000` show this is 10-20% slower for unquote and unquote_to_bytes and no different for typical inputs that are short or lack much unicode or % escaping. But the functions are already quite fast anyways so not a big deal. The slowdown scales consistently linear with input size as expected. Memory usage observed manually using `/usr/bin/time -v` on `python -m timeit` runs of larger inputs. Unittesting memory consumption is difficult and does not seem worthwhile. Observed memory usage is ~1/2 for `unquote()` and <1/3 for `unquote_to_bytes()` using `python -m timeit -s 'from urllib.parse import unquote, unquote_to_bytes; v="\u0141%01\u0161%20"*500_000' 'unquote_to_bytes(v)'` as a test.
* gh-98778: Update HTTPError to initialize properly even if fp is None (gh-99966)Dong-hee Na2022-12-081-7/+4
|
* bpo-45975: Simplify some while-loops with walrus operator (GH-29347)Nick Drozd2022-11-261-8/+2
|
* gh-99418: Make urllib.parse.urlparse enforce that a scheme must begin with ↵Ben Kallus2022-11-131-1/+1
| | | | | | | | | | | an alphabetical ASCII character. (#99421) Prevent urllib.parse.urlparse from accepting schemes that don't begin with an alphabetical ASCII character. RFC 3986 defines a scheme like this: `scheme = ALPHA *( ALPHA / DIGIT / "+" / "-" / "." )` RFC 2234 defines an ALPHA like this: `ALPHA = %x41-5A / %x61-7A` The WHATWG URL spec defines a scheme like this: `"A URL-scheme string must be one ASCII alpha, followed by zero or more of ASCII alphanumeric, U+002B (+), U+002D (-), and U+002E (.)."`
* gh-96035: Make urllib.parse.urlparse reject non-numeric ports (#98273)Ben Kallus2022-10-201-9/+8
| | | Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
* bpo-43564: preserve original exception in args of FTP URLError (#24938)Carl Meyer2022-10-101-1/+1
| | | | | | | * bpo-43564: preserve original error in args of FTP URLError * Add NEWS blurb Co-authored-by: Carl Meyer <carljm@instagram.com>
* gh-91539: improve performance of get_proxies_environment (#91566)Pieter Eendebak2022-10-051-10/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * improve performance of get_proxies_environment when there are many environment variables * 📜🤖 Added by blurb_it. * fix case of short env name * fix formatting * fix whitespace * whitespace * Update Lib/urllib/request.py Co-authored-by: Carl Meyer <carl@oddbird.net> * Update Lib/urllib/request.py Co-authored-by: Carl Meyer <carl@oddbird.net> * Update Lib/urllib/request.py Co-authored-by: Carl Meyer <carl@oddbird.net> * Update Lib/urllib/request.py Co-authored-by: Carl Meyer <carl@oddbird.net> * whitespace * Update Misc/NEWS.d/next/Library/2022-04-15-11-29-38.gh-issue-91539.7WgVuA.rst Co-authored-by: Carl Meyer <carl@oddbird.net> * Update Lib/urllib/request.py Co-authored-by: Carl Meyer <carl@oddbird.net> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Carl Meyer <carl@oddbird.net>
* gh-95865: Further reduce quote_from_bytes memory consumption (#96860)Gregory P. Smith2022-09-191-1/+9
| | | on large input values. Based on Dennis Sweeney's chunking idea.
* gh-95865: Speed up urllib.parse.quote_from_bytes() (GH-95872)Dennis Sweeney2022-08-311-1/+1
|
* gh-94172: urllib.request avoids deprecated key_file/cert_file (#94232)Victor Stinner2022-06-261-3/+11
| | | | The urllib.request module no longer uses the deprecated key_file and cert_file parameter of the http.client module.
* gh-94172: urllib.request avoids deprecated check_hostname (#94193)Victor Stinner2022-06-241-2/+6
| | | | | | | | | | | The urllib.request no longer uses the deprecated check_hostname parameter of the http.client module. Add private http.client._create_https_context() helper to http.client, used by urllib.request. Remove the now redundant check on check_hostname and verify_mode in http.client: the SSLContext.check_hostname setter already implements the check.
* gh-84623: Remove unused imports in stdlib (#93773)Victor Stinner2022-06-132-2/+0
|
* bpo-42627: Fix incorrect parsing of Windows registry proxy settings (GH-26307)狂男风2022-05-111-16/+20
|
* Replace with_traceback() with exception chaining and reraising (GH-32074)Oleg Iarygin2022-03-302-9/+6
|
* bpo-46756: Fix authorization check in urllib.request (GH-31353)Serhiy Storchaka2022-02-251-4/+4
| | | | | | Fix a bug in urllib.request.HTTPPasswordMgr.find_user_password() and urllib.request.HTTPPasswordMgrWithPriorAuth.is_authenticated() which allowed to bypass authorization. For example, access to URI "example.org/foobar" was allowed if the user was authorized for URI "example.org/foo".
* bpo-45874: Handle empty query string correctly in urllib.parse.parse_qsl ↵Christian Sattler2021-12-121-2/+3
| | | | (#29716)
* bpo-40321: Add missing test, slightly expand documentation (GH-28760)Łukasz Langa2021-10-061-1/+1
|
* bpo-40321: Support HTTP response status code 308 in urllib.request (#19588)Jochem Schulenklopper2021-10-061-4/+11
| | | | | | | | | | | * Support HTTP response status code 308 in urllib. HTTP response status code 308 is defined in https://tools.ietf.org/html/rfc7538 to be the permanent redirect variant of 307 (temporary redirect). * Update documentation to include http_error_308() * Add blurb for bpo-40321 fix Co-authored-by: Roland Crosby <roland@rolandcrosby.com>
* Update URLs in comments and metadata to use HTTPS (GH-27458)Noah Kantrowitz2021-07-301-1/+1
|
* bpo-44002: Switch to lru_cache in urllib.parse. (GH-25798)Gregory P. Smith2021-05-121-29/+29
| | | | | | | | | | | | Switch to lru_cache in urllib.parse. urllib.parse now uses functool.lru_cache for its internal URL splitting and quoting caches instead of rolling its own like its the 90s. The undocumented internal Quoted class API is now deprecated as it had no reason to be public and no existing OSS users were found. The clear_cache() API remains undocumented but gets an explicit test as it is used in a few projects' (twisted, gevent) tests as well as our own regrtest.
* bpo-43882 Remove the newline, and tab early. From query and fragments. ↵Senthil Kumaran2021-05-051-3/+5
| | | | (GH-25921)
* bpo-43979: Remove unnecessary operation from urllib.parse.parse_qsl (GH-25756)Dong-hee Na2021-04-301-2/+1
| | | Automerge-Triggered-By: GH:gpshead
* bpo-43882 - urllib.parse should sanitize urls containing ASCII newline and ↵Senthil Kumaran2021-04-291-0/+6
| | | | | | | | tabs. (GH-25595) * issue43882 - urllib.parse should sanitize urls containing ASCII newline and tabs. Co-authored-by: Gregory P. Smith <greg@krypto.org> Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
* bpo-42967: coerce bytes separator to string in urllib.parse_qs(l) (#24818)Ken Jin2021-04-111-0/+1
| | | | | | | * coerce bytes separator to string * Add news * Update Misc/NEWS.d/next/Library/2021-03-11-00-31-41.bpo-42967.2PeQRw.rst
* bpo-43075: Fix ReDoS in urllib AbstractBasicAuthHandler (GH-24391)Yeting Li2021-04-071-1/+1
| | | | | | | Fix Regular Expression Denial of Service (ReDoS) vulnerability in urllib.request.AbstractBasicAuthHandler. The ReDoS-vulnerable regex has quadratic worst-case complexity and it allows cause a denial of service when identifying crafted invalid RFCs. This ReDoS issue is on the client side and needs remote attackers to control the HTTP server.
* bpo-42967: Fix urllib.parse docs and make logic clearer (GH-24536)Ken Jin2021-02-151-2/+1
|
* bpo-42967: only use '&' as a query string separator (#24297)Adam Goldschmidt2021-02-141-5/+15
| | | | | | | | | | | bpo-42967: [security] Address a web cache-poisoning issue reported in urllib.parse.parse_qsl(). urllib.parse will only us "&" as query string separator by default instead of both ";" and "&" as allowed in earlier versions. An optional argument seperator with default value "&" is added to specify the separator. Co-authored-by: Éric Araujo <merwok@netwok.org> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com> Co-authored-by: Éric Araujo <merwok@netwok.org>
* Allow / character in username,password fields in _PROXY envvars. (#23973)Senthil Kumaran2020-12-291-1/+5
|
* bpo-40968: Send http/1.1 ALPN extension (#20959)Christian Heimes2020-11-131-0/+2
| | | Signed-off-by: Christian Heimes <christian@python.org>
* bpo-41471: Ignore invalid prefix lengths in system proxy settings on macOS ↵Ronald Oussoren2020-10-191-0/+5
| | | | (GH-22762)
* bpo-39481: PEP 585 for a variety of modules (GH-19423)Batuhan Taşkaya2020-04-101-0/+3
| | | | | | | | | | - concurrent.futures - ctypes - http.cookies - multiprocessing - queue - tempfile - unittest.case - urllib.parse
* bpo-39503: CVE-2020-8492: Fix AbstractBasicAuthHandler (GH-18284)Victor Stinner2020-04-021-19/+50
| | | | | | | | | | | | | The AbstractBasicAuthHandler class of the urllib.request module uses an inefficient regular expression which can be exploited by an attacker to cause a denial of service. Fix the regex to prevent the catastrophic backtracking. Vulnerability reported by Ben Caller and Matt Schwager. AbstractBasicAuthHandler of urllib.request now parses all WWW-Authenticate HTTP headers and accepts multiple challenges per header: use the realm of the first Basic challenge. Co-Authored-By: Serhiy Storchaka <storchaka@gmail.com>
* bpo-39548: Fix handling of 'WWW-Authenticate' header for Digest Auth (GH-18338)Stephen Balousek2020-02-291-3/+3
| | | | | | | | | | | | | | | | | | | | | | * bpo-39548: Fix handling of 'WWW-Authenticate' header for Digest authentication - The 'qop' value in the 'WWW-Authenticate' header is optional. The presence of 'qop' in the header should be checked before its value is parsed with 'split'. Signed-off-by: Stephen Balousek <stephen@balousek.net> * bpo-39548: Fix handling of 'WWW-Authenticate' header for Digest authentication - Add NEWS item Signed-off-by: Stephen Balousek <stephen@balousek.net> * Update Misc/NEWS.d/next/Library/2020-02-06-05-33-52.bpo-39548.DF4FFe.rst Co-Authored-By: Brandt Bucher <brandtbucher@gmail.com> Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>
* bpo-37970: update and improve urlparse and urlsplit doc-strings (GH-16458)idomic2020-02-161-6/+35
|
* bpo-39057: Fix urllib.request.proxy_bypass_environment(). (GH-17619)Serhiy Storchaka2020-01-052-13/+15
| | | Ignore leading dots and no longer ignore a trailing newline.
* bpo-38686: fix HTTP Digest handling in request.py (#17045)PypeBros2019-11-221-2/+4
| | | | | | | | | | | | | | | | | | | | | * fix HTTP Digest handling in request.py There is a bug triggered when server replies to a request with `WWW-Authenticate: Digest` where `qop="auth,auth-int"` rather than mere `qop="auth"`. Having both `auth` and `auth-int` is legitimate according to the `qop-options` rule in §3.2.1 of [[https://www.ietf.org/rfc/rfc2617.txt|RFC 2617]]: > qop-options = "qop" "=" <"> 1#qop-value <"> > qop-value = "auth" | "auth-int" | token > **qop-options**: [...] If present, it is a quoted string **of one or more** tokens indicating the "quality of protection" values supported by the server. The value `"auth"` indicates authentication; the value `"auth-int"` indicates authentication with integrity protection This is description confirmed by the definition of the [//n//]`#`[//m//]//rule// extended-BNF pattern defined in §2.1 of [[https://www.ietf.org/rfc/rfc2616.txt|RFC 2616]] as 'a comma-separated list of //rule// with at least //n// and at most //m// items'. When this reply is parsed by `get_authorization`, request.py only tests for identity with `'auth'`, failing to recognize it as one of the supported modes the server announced, and claims that `"qop 'auth,auth-int' is not supported"`. * 📜🤖 Added by blurb_it. * bpo-38686 review fix: remember why. * fix trailing space in Lib/urllib/request.py Co-Authored-By: Brandt Bucher <brandtbucher@gmail.com>
generated by cgit v0.12 at 2025-03-27 01:31:32 (GMT)