summaryrefslogtreecommitdiffstats
path: root/Doc/ref2.tex
blob: 250bd2e771957a9242579bb17e99d60c5ece5090 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
\chapter{Lexical analysis}

A Python program is read by a {\em parser}.  Input to the parser is a
stream of {\em tokens}, generated by the {\em lexical analyzer}.  This
chapter describes how the lexical analyzer breaks a file into tokens.
\index{lexical analysis}
\index{parser}
\index{token}

\section{Line structure}

A Python program is divided in a number of logical lines.  The end of
a logical line is represented by the token NEWLINE.  Statements cannot
cross logical line boundaries except where NEWLINE is allowed by the
syntax (e.g. between statements in compound statements).
\index{line structure}
\index{logical line}
\index{NEWLINE token}

\subsection{Comments}

A comment starts with a hash character (\verb\#\) that is not part of
a string literal, and ends at the end of the physical line.  A comment
always signifies the end of the logical line.  Comments are ignored by
the syntax.
\index{comment}
\index{logical line}
\index{physical line}
\index{hash character}

\subsection{Line joining}

Two or more physical lines may be joined into logical lines using
backslash characters (\verb/\/), as follows: when a physical line ends
in a backslash that is not part of a string literal or comment, it is
joined with the following forming a single logical line, deleting the
backslash and the following end-of-line character.  For example:
\index{physical line}
\index{line joining}
\index{backslash character}
%
\begin{verbatim}
month_names = ['Januari', 'Februari', 'Maart',     \
               'April',   'Mei',      'Juni',      \
               'Juli',    'Augustus', 'September', \
               'Oktober', 'November', 'December']
\end{verbatim}

\subsection{Blank lines}

A logical line that contains only spaces, tabs, and possibly a
comment, is ignored (i.e., no NEWLINE token is generated), except that
during interactive input of statements, an entirely blank logical line
terminates a multi-line statement.
\index{blank line}

\subsection{Indentation}

Leading whitespace (spaces and tabs) at the beginning of a logical
line is used to compute the indentation level of the line, which in
turn is used to determine the grouping of statements.
\index{indentation}
\index{whitespace}
\index{leading whitespace}
\index{space}
\index{tab}
\index{grouping}
\index{statement grouping}

First, tabs are replaced (from left to right) by one to eight spaces
such that the total number of characters up to there is a multiple of
eight (this is intended to be the same rule as used by {\UNIX}).  The
total number of spaces preceding the first non-blank character then
determines the line's indentation.  Indentation cannot be split over
multiple physical lines using backslashes.

The indentation levels of consecutive lines are used to generate
INDENT and DEDENT tokens, using a stack, as follows.
\index{INDENT token}
\index{DEDENT token}

Before the first line of the file is read, a single zero is pushed on
the stack; this will never be popped off again.  The numbers pushed on
the stack will always be strictly increasing from bottom to top.  At
the beginning of each logical line, the line's indentation level is
compared to the top of the stack.  If it is equal, nothing happens.
If it is larger, it is pushed on the stack, and one INDENT token is
generated.  If it is smaller, it {\em must} be one of the numbers
occurring on the stack; all numbers on the stack that are larger are
popped off, and for each number popped off a DEDENT token is
generated.  At the end of the file, a DEDENT token is generated for
each number remaining on the stack that is larger than zero.

Here is an example of a correctly (though confusingly) indented piece
of Python code:

\begin{verbatim}
def perm(l):
        # Compute the list of all permutations of l

    if len(l) <= 1:
                  return [l]
    r = []
    for i in range(len(l)):
             s = l[:i] + l[i+1:]
             p = perm(s)
             for x in p:
              r.append(l[i:i+1] + x)
    return r
\end{verbatim}

The following example shows various indentation errors:

\begin{verbatim}
    def perm(l):                        # error: first line indented
    for i in range(len(l)):             # error: not indented
        s = l[:i] + l[i+1:]
            p = perm(l[:i] + l[i+1:])   # error: unexpected indent
            for x in p:
                    r.append(l[i:i+1] + x)
                return r                # error: inconsistent dedent
\end{verbatim}

(Actually, the first three errors are detected by the parser; only the
last error is found by the lexical analyzer --- the indentation of
\verb\return r\ does not match a level popped off the stack.)

\section{Other tokens}

Besides NEWLINE, INDENT and DEDENT, the following categories of tokens
exist: identifiers, keywords, literals, operators, and delimiters.
Spaces and tabs are not tokens, but serve to delimit tokens.  Where
ambiguity exists, a token comprises the longest possible string that
forms a legal token, when read from left to right.

\section{Identifiers}

Identifiers (also referred to as names) are described by the following
lexical definitions:
\index{identifier}
\index{name}

\begin{verbatim}
identifier:     (letter|"_") (letter|digit|"_")*
letter:         lowercase | uppercase
lowercase:      "a"..."z"
uppercase:      "A"..."Z"
digit:          "0"..."9"
\end{verbatim}

Identifiers are unlimited in length.  Case is significant.

\subsection{Keywords}

The following identifiers are used as reserved words, or {\em
keywords} of the language, and cannot be used as ordinary
identifiers.  They must be spelled exactly as written here:
\index{keyword}
\index{reserved word}

\begin{verbatim}
and        del        for        in         print
break      elif       from       is         raise
class      else       global     not        return
continue   except     if         or         try
def        finally    import     pass       while
\end{verbatim}

%	# This Python program sorts and formats the above table
%	import string
%	l = []
%	try:
%		while 1:
%			l = l + string.split(raw_input())
%	except EOFError:
%		pass
%	l.sort()
%	for i in range((len(l)+4)/5):
%		for j in range(i, len(l), 5):
%			print string.ljust(l[j], 10),
%		print

\section{Literals} \label{literals}

Literals are notations for constant values of some built-in types.
\index{literal}
\index{constant}

\subsection{String literals}

String literals are described by the following lexical definitions:
\index{string literal}

\begin{verbatim}
stringliteral:  "'" stringitem* "'"
stringitem:     stringchar | escapeseq
stringchar:     <any ASCII character except newline or "\" or "'">
escapeseq:      "'" <any ASCII character except newline>
\end{verbatim}
\index{ASCII}

String literals cannot span physical line boundaries.  Escape
sequences in strings are actually interpreted according to rules
similar to those used by Standard C.  The recognized escape sequences
are:
\index{physical line}
\index{escape sequence}
\index{Standard C}
\index{C}

\begin{center}
\begin{tabular}{|l|l|}
\hline
\verb/\\/	& Backslash (\verb/\/) \\
\verb/\'/	& Single quote (\verb/'/) \\
\verb/\a/	& ASCII Bell (BEL) \\
\verb/\b/	& ASCII Backspace (BS) \\
%\verb/\E/	& ASCII Escape (ESC) \\
\verb/\f/	& ASCII Formfeed (FF) \\
\verb/\n/	& ASCII Linefeed (LF) \\
\verb/\r/	& ASCII Carriage Return (CR) \\
\verb/\t/	& ASCII Horizontal Tab (TAB) \\
\verb/\v/	& ASCII Vertical Tab (VT) \\
\verb/\/{\em ooo}	& ASCII character with octal value {\em ooo} \\
\verb/\x/{\em xx...}	& ASCII character with hex value {\em xx...} \\
\hline
\end{tabular}
\end{center}
\index{ASCII}

In strict compatibility with Standard C, up to three octal digits are
accepted, but an unlimited number of hex digits is taken to be part of
the hex escape (and then the lower 8 bits of the resulting hex number
are used in all current implementations...).

All unrecognized escape sequences are left in the string unchanged,
i.e., {\em the backslash is left in the string.}  (This behavior is
useful when debugging: if an escape sequence is mistyped, the
resulting output is more easily recognized as broken.  It also helps a
great deal for string literals used as regular expressions or
otherwise passed to other modules that do their own escape handling.)
\index{unrecognized escape sequence}

\subsection{Numeric literals}

There are three types of numeric literals: plain integers, long
integers, and floating point numbers.
\index{number}
\index{numeric literal}
\index{integer literal}
\index{plain integer literal}
\index{long integer literal}
\index{floating point literal}
\index{hexadecimal literal}
\index{octal literal}
\index{decimal literal}

Integer and long integer literals are described by the following
lexical definitions:

\begin{verbatim}
longinteger:    integer ("l"|"L")
integer:        decimalinteger | octinteger | hexinteger
decimalinteger: nonzerodigit digit* | "0"
octinteger:     "0" octdigit+
hexinteger:     "0" ("x"|"X") hexdigit+

nonzerodigit:   "1"..."9"
octdigit:       "0"..."7"
hexdigit:        digit|"a"..."f"|"A"..."F"
\end{verbatim}

Although both lower case `l' and upper case `L' are allowed as suffix
for long integers, it is strongly recommended to always use `L', since
the letter `l' looks too much like the digit `1'.

Plain integer decimal literals must be at most $2^{31} - 1$ (i.e., the
largest positive integer, assuming 32-bit arithmetic).  Plain octal and
hexadecimal literals may be as large as $2^{32} - 1$, but values
larger than $2^{31} - 1$ are converted to a negative value by
subtracting $2^{32}$.  There is no limit for long integer literals.

Some examples of plain and long integer literals:

\begin{verbatim}
7     2147483647                        0177    0x80000000
3L    79228162514264337593543950336L    0377L   0x100000000L
\end{verbatim}

Floating point literals are described by the following lexical
definitions:

\begin{verbatim}
floatnumber:    pointfloat | exponentfloat
pointfloat:     [intpart] fraction | intpart "."
exponentfloat:  (intpart | pointfloat) exponent
intpart:        digit+
fraction:       "." digit+
exponent:       ("e"|"E") ["+"|"-"] digit+
\end{verbatim}

The allowed range of floating point literals is
implementation-dependent.

Some examples of floating point literals:

\begin{verbatim}
3.14    10.    .001    1e100    3.14e-10
\end{verbatim}

Note that numeric literals do not include a sign; a phrase like
\verb\-1\ is actually an expression composed of the operator
\verb\-\ and the literal \verb\1\.

\section{Operators}

The following tokens are operators:
\index{operators}

\begin{verbatim}
+       -       *       /       %
<<      >>      &       |       ^       ~
<       ==      >       <=      <>      !=      >=
\end{verbatim}

The comparison operators \verb\<>\ and \verb\!=\ are alternate
spellings of the same operator.

\section{Delimiters}

The following tokens serve as delimiters or otherwise have a special
meaning:
\index{delimiters}

\begin{verbatim}
(       )       [       ]       {       }
;       ,       :       .       `       =
\end{verbatim}

The following printing ASCII characters are not used in Python.  Their
occurrence outside string literals and comments is an unconditional
error:
\index{ASCII}

\begin{verbatim}
@       $       "       ?
\end{verbatim}

They may be used by future versions of the language though!