From a8f6f1e2d63b90549278b1162d264bb279459007 Mon Sep 17 00:00:00 2001 From: Ezio Melotti Date: Sun, 20 Dec 2009 12:24:57 +0000 Subject: Fixed capitalization and markup; added imports in the example --- Doc/library/tokenize.rst | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst index deb7826..d66b07c 100644 --- a/Doc/library/tokenize.rst +++ b/Doc/library/tokenize.rst @@ -94,16 +94,19 @@ function it uses to do this is available: (as a string) and a list of any lines (not decoded from bytes) it has read in. - It detects the encoding from the presence of a utf-8 bom or an encoding - cookie as specified in pep-0263. If both a bom and a cookie are present, + It detects the encoding from the presence of a UTF-8 BOM or an encoding + cookie as specified in :pep:`263`. If both a BOM and a cookie are present, but disagree, a SyntaxError will be raised. - If no encoding is specified, then the default of 'utf-8' will be returned. + If no encoding is specified, then the default of ``'utf-8'`` will be returned. Example of a script re-writer that transforms float literals into Decimal objects:: + from tokenize import tokenize, untokenize, NUMBER, STRING, NAME, OP + from io import BytesIO + def decistmt(s): """Substitute Decimals for floats in a string of statements. -- cgit v0.12