diff options
| author | Lysandros Nikolaou <lisandrosnik@gmail.com> | 2022-10-06 23:07:17 (GMT) |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2022-10-06 23:07:17 (GMT) |
| commit | cbf0afd8a1474d68310331af9218606959d4cc22 (patch) | |
| tree | cd421653d73d25c89fa8c02e5819517efa76f7b3 /Parser/pegen_errors.c | |
| parent | b9d2e8171696514e9226164005f7bf24bf69e66d (diff) | |
| download | cpython-cbf0afd8a1474d68310331af9218606959d4cc22.zip cpython-cbf0afd8a1474d68310331af9218606959d4cc22.tar.gz cpython-cbf0afd8a1474d68310331af9218606959d4cc22.tar.bz2 | |
gh-97973: Return all necessary information from the tokenizer (GH-97984)
Right now, the tokenizer only returns type and two pointers to the start and end of the token.
This PR modifies the tokenizer to return the type and set all of the necessary information,
so that the parser does not have to this.
Diffstat (limited to 'Parser/pegen_errors.c')
| -rw-r--r-- | Parser/pegen_errors.c | 5 |
1 files changed, 2 insertions, 3 deletions
diff --git a/Parser/pegen_errors.c b/Parser/pegen_errors.c index 95bbd43..7738cba 100644 --- a/Parser/pegen_errors.c +++ b/Parser/pegen_errors.c @@ -164,11 +164,10 @@ _PyPegen_tokenize_full_source_to_check_for_errors(Parser *p) { Py_ssize_t current_err_line = current_token->lineno; int ret = 0; + struct token new_token; for (;;) { - const char *start; - const char *end; - switch (_PyTokenizer_Get(p->tok, &start, &end)) { + switch (_PyTokenizer_Get(p->tok, &new_token)) { case ERRORTOKEN: if (p->tok->level != 0) { int error_lineno = p->tok->parenlinenostack[p->tok->level-1]; |