summaryrefslogtreecommitdiffstats
path: root/Include/internal
Commit message (Collapse)AuthorAgeFilesLines
* [3.10] gh-95778: Correctly pre-check for int-to-str conversion (GH-96537) ↵Gregory P. Smith2022-09-041-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (#96563) Converting a large enough `int` to a decimal string raises `ValueError` as expected. However, the raise comes _after_ the quadratic-time base-conversion algorithm has run to completion. For effective DOS prevention, we need some kind of check before entering the quadratic-time loop. Oops! =) The quick fix: essentially we catch _most_ values that exceed the threshold up front. Those that slip through will still be on the small side (read: sufficiently fast), and will get caught by the existing check so that the limit remains exact. The justification for the current check. The C code check is: ```c max_str_digits / (3 * PyLong_SHIFT) <= (size_a - 11) / 10 ``` In GitHub markdown math-speak, writing $M$ for `max_str_digits`, $L$ for `PyLong_SHIFT` and $s$ for `size_a`, that check is: $$\left\lfloor\frac{M}{3L}\right\rfloor \le \left\lfloor\frac{s - 11}{10}\right\rfloor$$ From this it follows that $$\frac{M}{3L} < \frac{s-1}{10}$$ hence that $$\frac{L(s-1)}{M} > \frac{10}{3} > \log_2(10).$$ So $$2^{L(s-1)} > 10^M.$$ But our input integer $a$ satisfies $|a| \ge 2^{L(s-1)}$, so $|a|$ is larger than $10^M$. This shows that we don't accidentally capture anything _below_ the intended limit in the check. <!-- gh-issue-number: gh-95778 --> * Issue: gh-95778 <!-- /gh-issue-number --> Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org> (cherry picked from commit b126196838bbaf5f4d35120e0e6bcde435b0b480) Co-authored-by: Mark Dickinson <dickinsm@gmail.com>
* [3.10] gh-95778: CVE-2020-10735: Prevent DoS by very large int() (#96501)Gregory P. Smith2022-09-023-0/+39
| | | | | | | | | | | | | | | | | Integer to and from text conversions via CPython's bignum `int` type is not safe against denial of service attacks due to malicious input. Very large input strings with hundred thousands of digits can consume several CPU seconds. This PR comes fresh from a pile of work done in our private PSRT security response team repo. This backports https://github.com/python/cpython/pull/96499 aka 511ca9452033ef95bc7d7fc404b8161068226002 Signed-off-by: Christian Heimes [Red Hat] <christian@python.org> Tons-of-polishing-up-by: Gregory P. Smith [Google] <greg@krypto.org> Reviews via the private PSRT repo via many others (see the NEWS entry in the PR). <!-- gh-issue-number: gh-95778 --> * Issue: gh-95778 <!-- /gh-issue-number --> I wrote up [a one pager for the release managers](https://docs.google.com/document/d/1KjuF_aXlzPUxTK4BMgezGJ2Pn7uevfX7g0_mvgHlL7Y/edit#).
* Fix typo in internal/pycore_atomic.h (GH-95939)Miss Islington (bot)2022-08-131-1/+1
| | | | | (cherry picked from commit 8281cbddc6f0fbc94f0c21cacfac79a2d4057a4b) Co-authored-by: fluesvamp <105884371+fluesvamp@users.noreply.github.com>
* gh-93065: Fix HAMT to iterate correctly over 7-level deep trees (GH-93066) ↵Miss Islington (bot)2022-05-241-1/+13
| | | | | | | | | | | | (GH-93146) Also while there, clarify a few things about why we reduce the hash to 32 bits. Co-authored-by: Eli Libman <eli@hyro.ai> Co-authored-by: Yury Selivanov <yury@edgedb.com> Co-authored-by: Łukasz Langa <lukasz@langa.pl> (cherry picked from commit c1f5c903a7e4ed27190488f4e33b00d3c3d952e5)
* [3.10] bpo-46006: Revert "bpo-40521: Per-interpreter interned strings ↵Victor Stinner2022-01-061-9/+3
| | | | | | | | | | | | | | | | | (GH-20085)" (GH-30422) (GH-30425) This reverts commit ea251806b8dffff11b30d2182af1e589caf88acf. Keep "assert(interned == NULL);" in _PyUnicode_Fini(), but only for the main interpreter. Keep _PyUnicode_ClearInterned() changes avoiding the creation of a temporary Python list object. Leave the PyInterpreterState structure unchanged to keep the ABI backward compatibility with Python 3.10.0: rename the "interned" member to "unused_interned". (cherry picked from commit 35d6540c904ef07b8602ff014e520603f84b5886)
* bpo-46042: Improve SyntaxError locations in the symbol table (GH-30059) ↵Miss Islington (bot)2021-12-121-2/+9
| | | | | | | | | (GH-30064) (cherry picked from commit 59435eea08d30796174552c0ca03c59b41adf8a5) Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com> Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
* [3.10] Fix typos in the Include directory (GH-28745) (GH-28789)Christian Clauss2021-10-071-1/+1
| | | (cherry picked from commit 8e8f7522171ef82f2f5049940f815e00e38c6f42)
* [3.10] Remove trailing spaces (GH-28709)Serhiy Storchaka2021-10-031-1/+1
|
* [3.10] [codemod] Fix non-matching bracket pairs (GH-28473) (GH-28511)Łukasz Langa2021-09-211-1/+1
| | | | | | | | Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu> Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> Co-authored-by: Łukasz Langa <lukasz@langa.pl> (cherry picked from commit 8f943ca25732d548cf9f0b0393ba8d582fb93e29) Co-authored-by: Mohamad Mansour <66031317+mohamadmansourX@users.noreply.github.com>
* [3.10] bpo-45083: Include the exception class qualname when formatting an ↵Miss Islington (bot)2021-09-081-0/+2
| | | | | | | | | | | | exception (GH-28119) (GH-28134) Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@innova.no> (cherry picked from commit b4b6342848ec0459182a992151099252434cc619) Co-authored-by: Irit Katriel <1055913+iritkatriel@users.noreply.github.com> * Use a private version of _PyType_GetQualName Co-authored-by: Łukasz Langa <lukasz@langa.pl>
* [3.10] bpo-44654: Do not export the union type related symbols (GH-27223). ↵Serhiy Storchaka2021-07-181-3/+3
| | | | | | (GH-27225) (cherry picked from commit 8f50f44592190b5a8cb115f0d58d577036e68308)
* [3.10] bpo-44654: Refactor and clean up the union type implementation ↵Miss Islington (bot)2021-07-171-3/+4
| | | | | | (GH-27196) (GH-27219) (cherry picked from commit 0fd27375cabd12e68a2f12cfeca11a2d5043429e)
* [3.10] bpo-44490: Add __parameters__ and __getitem__ to types.Union ↵Serhiy Storchaka2021-07-171-0/+3
| | | | | | | | | | | (GH-26980) (GH-27207) Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com> Co-authored-by: Guido van Rossum <gvanrossum@gmail.com>. (cherry picked from commit c45fa1a5d9b419cf13ad4b5a7cb453956495b83e) Co-authored-by: Yurii Karabas <1998uriyyo@gmail.com> Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com>
* bpo-44458: Ensure BUFFER_BLOCK_SIZE symbol is statically allocated. ↵Miss Islington (bot)2021-06-221-1/+1
| | | | | | | | (GH-26808) (GH-26844) * bpo-44458: Ensure BUFFER_BLOCK_SIZE symbol is statically allocated. (cherry picked from commit 92c2e91580521ba5c85aa3205a0211df5b48689b) Co-authored-by: Russell Keith-Magee <russell@keith-magee.com>
* bpo-44184: Fix subtype_dealloc() for freed type (GH-26274) (GH-26290)Miss Islington (bot)2021-05-211-1/+1
| | | | | | | | | | | | | | Fix a crash at Python exit when a deallocator function removes the last strong reference to a heap type. Don't read type memory after calling basedealloc() since basedealloc() can deallocate the type and free its memory. _PyMem_IsPtrFreed() argument is now constant. (cherry picked from commit 615069eb08494d089bf24e43547fbc482ed699b8) Co-authored-by: Victor Stinner <vstinner@python.org> Co-authored-by: Victor Stinner <vstinner@python.org>
* bpo-38530: Refactor and improve AttributeError suggestions (GH-25776)Dennis Sweeney2021-05-031-0/+2
| | | | | | | | | | | | | | | | | | | | - Make case-swaps half the cost of any other edit - Refactor Levenshtein code to not use memory allocator, and to bail early on no match. - Add comments to Levenshtein distance code - Add test cases for Levenshtein distance behind a debug macro - Set threshold to `(name_size + item_size + 3) * MOVE_COST / 6`. - Reasoning: similar to `difflib.SequenceMatcher.ratio()` >= 2/3: ``` "Multiset Jaccard similarity" >= 2/3 matching letters / total letters >= 2/3 (name_size - distance + item_size - distance) / (name_size + item_size) >= 2/3 1 - (2*distance) / (name_size + item_size) >= 2/3 1/3 >= (2*distance) / (name_size + item_size) (name_size + item_size) / 6 >= distance With rounding: (name_size + item_size + 3) // 6 >= distance ``` Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
* bpo-43916: Move the _PyStructSequence_InitType function to the internal API ↵Pablo Galindo2021-05-031-0/+21
| | | | (GH-25854)
* bpo-42725: Render annotations effectless on symbol table with PEP 563 (GH-25583)Batuhan Taskaya2021-05-031-1/+1
|
* bpo-43667: Fix broken Unicode encoding in non-UTF locales on Solaris (GH-25096)Jakub Kulík2021-04-301-0/+12
|
* bpo-43977: Use tp_flags for collection matching (GH-25723)Mark Shannon2021-04-301-4/+0
| | | | | | | | | | | | | * Add Py_TPFLAGS_SEQUENCE and Py_TPFLAGS_MAPPING, add to all relevant standard builtin classes. * Set relevant flags on collections.abc.Sequence and Mapping. * Use flags in MATCH_SEQUENCE and MATCH_MAPPING opcodes. * Inherit Py_TPFLAGS_SEQUENCE and Py_TPFLAGS_MAPPING. * Add NEWS * Remove interpreter-state map_abc and seq_abc fields.
* bpo-43892: Make match patterns explicit in the AST (GH-25585)Nick Coghlan2021-04-293-20/+100
| | | Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>
* bpo-43962: Fix _PyInterpreterState_IDIncref() (GH-25683)Victor Stinner2021-04-281-1/+1
| | | | _PyInterpreterState_IDIncref() now calls _PyInterpreterState_IDInitref() and always increments id_refcount.
* bpo-41486: Faster bz2/lzma/zlib via new output buffering (GH-21740)Ma Lin2021-04-281-0/+317
| | | | | | | | | Faster bz2/lzma/zlib via new output buffering. Also adds .readall() function to _compression.DecompressReader class to take best advantage of this in the consume-all-output at once scenario. Often a 5-20% speedup in common scenarios due to less data copying. Contributed by Ma Lin.
* bpo-42609: Check recursion depth in the AST validator and optimizer (GH-23744)Serhiy Storchaka2021-04-251-0/+3
|
* bpo-43914: Highlight invalid ranges in SyntaxErrors (#25525)Pablo Galindo2021-04-231-0/+2
| | | | | | | | | | | | | | | | | To improve the user experience understanding what part of the error messages associated with SyntaxErrors is wrong, we can highlight the whole error range and not only place the caret at the first character. In this way: >>> foo(x, z for z in range(10), t, w) File "<stdin>", line 1 foo(x, z for z in range(10), t, w) ^ SyntaxError: Generator expression must be parenthesized becomes >>> foo(x, z for z in range(10), t, w) File "<stdin>", line 1 foo(x, z for z in range(10), t, w) ^^^^^^^^^^^^^^^^^^^^ SyntaxError: Generator expression must be parenthesized
* bpo-40137: Add pycore_moduleobject.h internal header (GH-25507)Victor Stinner2021-04-211-0/+42
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add pycore_moduleobject.h internal header file with static inline functions to access module members: * _PyModule_GetDict() * _PyModule_GetDef() * _PyModule_GetState() These functions don't check at runtime if their argument has a valid type and can be inlined even if Python is not built with LTO. _PyType_GetModuleByDef() uses _PyModule_GetDef(). Replace PyModule_GetState() with _PyModule_GetState() in the extension modules, considered as performance sensitive: * _abc * _functools * _operator * _pickle * _queue * _random * _sre * _struct * _thread * _winapi * array * posix The following extensions are now built with the Py_BUILD_CORE_MODULE macro defined, to be able to use the internal pycore_moduleobject.h header: _abc, array, _operator, _queue, _sre, _struct.
* bpo-38530: Offer suggestions on AttributeError (#16856)Pablo Galindo2021-04-141-0/+2
| | | | | | | | | When printing AttributeError, PyErr_Display will offer suggestions of similar attribute names in the object that the exception was raised from: >>> collections.namedtoplo Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: module 'collections' has no attribute 'namedtoplo'. Did you mean: namedtuple?
* bpo-43760: Speed up check for tracing in interpreter dispatch (#25276)Mark Shannon2021-04-131-6/+0
| | | | | | | | | * Remove redundant tracing_possible field from interpreter state. * Move 'use_tracing' from tstate onto C stack, for fastest possible checking in dispatch logic. * Add comments stressing the importance stack discipline when dealing with CFrames. * Add NEWS
* bpo-43770: Refactor PyType_Ready() function (GH-25336)Victor Stinner2021-04-111-0/+4
| | | | | | * Split PyType_Ready() into sub-functions. * type_ready_mro() now checks if bases are static types earlier. * Check tp_name earlier, in type_ready_checks(). * Add _PyType_IsReady() macro to check if a type is ready.
* bpo-43798: Add source location attributes to alias (GH-25324)Matthew Suozzo2021-04-101-1/+7
| | | | | | | * Add source location attributes to alias. * Move alias star construction to pegen helper. Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-43244: Rename pycore_ast.h functions to _PyAST_xxx() (GH-25252)Victor Stinner2021-04-071-230/+171
| | | | | | Rename AST functions of pycore_ast.h to use the "_PyAST_" prefix. Remove macros creating aliases without prefix. For example, Module() becomes _PyAST_Module(). Update Grammar/python.gram to use _PyAST_xxx() functions.
* bpo-43244: Remove Yield macro from pycore_ast.h (GH-25243)Victor Stinner2021-04-071-3/+0
| | | | | | | | * pycore_ast.h no longer defines the Yield macro. * Fix a compiler warning on Windows: "warning C4005: 'Yield': macro redefinition". * Python-ast.c now defines directly functions with their real _Py_xxx() name, rather than xxx(). * Remove "#undef Yield" in C files including pycore_ast.h.
* bpo-43687: Py_Initialize() creates singletons earlier (GH-25147)Victor Stinner2021-04-022-4/+7
| | | | | Reorganize pycore_interp_init() to initialize singletons before the the first PyType_Ready() call. Fix an issue when Python is configured using --without-doc-strings.
* bpo-43510: Implement PEP 597 opt-in EncodingWarning. (GH-19481)Inada Naoki2021-03-291-0/+1
| | | | | | | | | | | See [PEP 597](https://www.python.org/dev/peps/pep-0597/). * Add `-X warn_default_encoding` and `PYTHONWARNDEFAULTENCODING`. * Add EncodingWarning * Add io.text_encoding() * open(), TextIOWrapper() emits EncodingWarning when encoding is omitted and warn_default_encoding is enabled. * _pyio.TextIOWrapper() uses UTF-8 as fallback default encoding used when failed to import locale module. (used during building Python) * bz2, configparser, gzip, lzma, pathlib, tempfile modules use io.text_encoding(). * What's new entry
* bpo-43244: Remove the pyarena.h header (GH-25007)Victor Stinner2021-03-243-4/+71
| | | | | | | | | | | | | | | | | | Remove the pyarena.h header file with functions: * PyArena_New() * PyArena_Free() * PyArena_Malloc() * PyArena_AddPyObject() These functions were undocumented, excluded from the limited C API, and were only used internally by the compiler. Add pycore_pyarena.h header. Rename functions: * PyArena_New() => _PyArena_New() * PyArena_Free() => _PyArena_Free() * PyArena_Malloc() => _PyArena_Malloc() * PyArena_AddPyObject() => _PyArena_AddPyObject()
* bpo-43244: Remove parser_interface.h header file (GH-25001)Victor Stinner2021-03-241-0/+31
| | | | | | | | | | | | | | | | | | | | | | | | | Remove parser functions using the "struct _mod" type, because the AST C API was removed: * PyParser_ASTFromFile() * PyParser_ASTFromFileObject() * PyParser_ASTFromFilename() * PyParser_ASTFromString() * PyParser_ASTFromStringObject() These functions were undocumented and excluded from the limited C API. Add pycore_parser.h internal header file. Rename functions: * PyParser_ASTFromFileObject() => _PyParser_ASTFromFile() * PyParser_ASTFromStringObject() => _PyParser_ASTFromString() These functions are no longer exported (replace PyAPI_FUNC() with extern). Remove also _PyPegen_run_parser_from_file() function. Update test_peg_generator to use _PyPegen_run_parser_from_file_pointer() instead.
* bpo-43244: Add pycore_compile.h header file (GH-25000)Victor Stinner2021-03-231-0/+40
| | | | | | | | | | | | | | | | | | | | | | | | | | | Remove the compiler functions using "struct _mod" type, because the public AST C API was removed: * PyAST_Compile() * PyAST_CompileEx() * PyAST_CompileObject() * PyFuture_FromAST() * PyFuture_FromASTObject() These functions were undocumented and excluded from the limited C API. Rename functions: * PyAST_CompileObject() => _PyAST_Compile() * PyFuture_FromASTObject() => _PyFuture_FromAST() Moreover, _PyFuture_FromAST() is no longer exported (replace PyAPI_FUNC() with extern). _PyAST_Compile() remains exported for test_peg_generator. Remove also compatibility functions: * PyAST_Compile() * PyAST_CompileEx() * PyFuture_FromAST()
* bpo-43244: Remove ast.h, asdl.h, Python-ast.h headers (GH-24933)Victor Stinner2021-03-233-4/+928
| | | | | | | | | | | | | | | | These functions were undocumented and excluded from the limited C API. Most names defined by these header files were not prefixed by "Py" and so could create names conflicts. For example, Python-ast.h defined a "Yield" macro which was conflict with the "Yield" name used by the Windows <winbase.h> header. Use the Python ast module instead. * Move Include/asdl.h to Include/internal/pycore_asdl.h. * Move Include/Python-ast.h to Include/internal/pycore_ast.h. * Remove ast.h header file. * pycore_symtable.h no longer includes Python-ast.h.
* Revert "bpo-40521: Make dtoa bigint free list per-interpreter (GH-24821)" ↵Victor Stinner2021-03-222-20/+0
| | | | | (GH-24964) This reverts commit 5bd1059184b154d339f1bd53d23c98b5bcf14c8c.
* bpo-43244: Remove symtable.h header file (GH-24910)Victor Stinner2021-03-191-0/+125
| | | | | | | | | | | | | | | | | | Rename Include/symtable.h to to Include/internal/pycore_symtable.h, don't export symbols anymore (replace PyAPI_FUNC and PyAPI_DATA with extern) and rename functions: * PyST_GetScope() to _PyST_GetScope() * PySymtable_BuildObject() to _PySymtable_Build() * PySymtable_Free() to _PySymtable_Free() Remove PySymtable_Build(), Py_SymtableString() and Py_SymtableStringObject() functions. The Py_SymtableString() function was part the stable ABI by mistake but it could not be used, since the symtable.h header file was excluded from the limited C API. The Python symtable module remains available and is unchanged.
* bpo-43244: Remove the PyAST_Validate() function (GH-24911)Victor Stinner2021-03-181-0/+2
| | | | | | | | | | | | Remove the PyAST_Validate() function. It is no longer possible to build a AST object (mod_ty type) with the public C API. The function was already excluded from the limited C API (PEP 384). Rename PyAST_Validate() function to _PyAST_Validate(), move it to the internal C API, and don't export it anymore (replace PyAPI_FUNC with extern). The function was added in bpo-12575 by the commit 832bfe2ebd5ecfa92031cd40c8b41835ba90487f.
* bpo-43244: Add pycore_ast.h header file (GH-24908)Victor Stinner2021-03-171-0/+25
| | | | | | | | Move _PyAST_GetDocString() and _PyAST_ExprAsUnicode() functions the internal C API: from Include/ast.h to a new Include/internal/pycore_ast.h header file. Don't export these functions anymore: replace PyAPI_FUNC() with extern. Remove also unused includes.
* bpo-43244: Rename pycore_ast.h to pycore_ast_state.h (GH-24907)Victor Stinner2021-03-172-4/+4
|
* bpo-40521: Make dtoa bigint free list per-interpreter (GH-24821)junyixie2021-03-132-0/+20
|
* bpo-43356: Allow passing a signal number to interrupt_main() (GH-24755)Antoine Pitrou2021-03-111-0/+16
| | | | Also introduce a new C API ``PyErr_SetInterruptEx(int signum)``.
* bpo-43311: Create GIL autoTSSkey ealier (GH-24819)Victor Stinner2021-03-101-1/+4
| | | | | | At Python startup, call _PyGILState_Init() before PyInterpreterState_New() which calls _PyThreadState_GET(). When Python is built using --with-experimental-isolated-subinterpreters, _PyThreadState_GET() uses autoTSSkey.
* bpo-37146: Move _PyEval_DeactivateOpCache() to the internal C API (GH-24786)Victor Stinner2021-03-081-0/+2
| | | Don't export the symbol anymore.
* bpo-42128: Structural Pattern Matching (PEP 634) (GH-22917)Brandt Bucher2021-02-262-1/+14
| | | | | Co-authored-by: Guido van Rossum <guido@python.org> Co-authored-by: Talin <viridia@gmail.com> Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
* bpo-42990: Functions inherit current builtins (GH-24564)Victor Stinner2021-02-201-1/+4
| | | | | | | | | | | | | | | | | | | | The types.FunctionType constructor now inherits the current builtins if the globals dictionary has no "__builtins__" key, rather than using {"None": None} as builtins: same behavior as eval() and exec() functions. Defining a function with "def function(...): ..." in Python is not affected, globals cannot be overriden with this syntax: it also inherits the current builtins. PyFrame_New(), PyEval_EvalCode(), PyEval_EvalCodeEx(), PyFunction_New() and PyFunction_NewWithQualName() now inherits the current builtins namespace if the globals dictionary has no "__builtins__" key. * Add _PyEval_GetBuiltins() function. * _PyEval_BuiltinsFromGlobals() now uses _PyEval_GetBuiltins() if builtins cannot be found in globals. * Add tstate parameter to _PyEval_BuiltinsFromGlobals().
* bpo-43268: Pass interp rather than tstate to internal functions (GH-24580)Victor Stinner2021-02-195-36/+36
| | | | | | | | | | | | | | | Pass the current interpreter (interp) rather than the current Python thread state (tstate) to internal functions which only use the interpreter. Modified functions: * _PyXXX_Fini() and _PyXXX_ClearFreeList() functions * _PyEval_SignalAsyncExc(), make_pending_calls() * _PySys_GetObject(), sys_set_object(), sys_set_object_id(), sys_set_object_str() * should_audit(), set_flags_from_config(), make_flags() * _PyAtExit_Call() * init_stdio_encoding() * etc.
generated by cgit v0.12 at 2024-11-25 12:28:43 (GMT)