summaryrefslogtreecommitdiffstats
path: root/Doc/tutorial/floatingpoint.rst
diff options
context:
space:
mode:
authorGeorg Brandl <georg@python.org>2008-08-10 12:16:45 (GMT)
committerGeorg Brandl <georg@python.org>2008-08-10 12:16:45 (GMT)
commitbae1b94d9e0a1d1e5264ce08be236efbfb649d60 (patch)
tree637c8fd2d23a1771795dfe516050adaa74350f76 /Doc/tutorial/floatingpoint.rst
parent3d1c7dec4af44134390addc25b002dfd22c4c3f8 (diff)
downloadcpython-bae1b94d9e0a1d1e5264ce08be236efbfb649d60.zip
cpython-bae1b94d9e0a1d1e5264ce08be236efbfb649d60.tar.gz
cpython-bae1b94d9e0a1d1e5264ce08be236efbfb649d60.tar.bz2
Remove long integer output.
Diffstat (limited to 'Doc/tutorial/floatingpoint.rst')
-rw-r--r--Doc/tutorial/floatingpoint.rst12
1 files changed, 6 insertions, 6 deletions
diff --git a/Doc/tutorial/floatingpoint.rst b/Doc/tutorial/floatingpoint.rst
index 150e8fb..c9408c9 100644
--- a/Doc/tutorial/floatingpoint.rst
+++ b/Doc/tutorial/floatingpoint.rst
@@ -173,24 +173,24 @@ and recalling that *J* has exactly 53 bits (is ``>= 2**52`` but ``< 2**53``),
the best value for *N* is 56::
>>> 2**52
- 4503599627370496L
+ 4503599627370496
>>> 2**53
- 9007199254740992L
+ 9007199254740992
>>> 2**56/10
- 7205759403792793L
+ 7205759403792794.0
That is, 56 is the only value for *N* that leaves *J* with exactly 53 bits. The
best possible value for *J* is then that quotient rounded::
>>> q, r = divmod(2**56, 10)
>>> r
- 6L
+ 6
Since the remainder is more than half of 10, the best approximation is obtained
by rounding up::
>>> q+1
- 7205759403792794L
+ 7205759403792794
Therefore the best possible approximation to 1/10 in 754 double precision is
that over 2\*\*56, or ::
@@ -211,7 +211,7 @@ If we multiply that fraction by 10\*\*30, we can see the (truncated) value of
its 30 most significant decimal digits::
>>> 7205759403792794 * 10**30 / 2**56
- 100000000000000005551115123125L
+ 100000000000000005551115123125
meaning that the exact number stored in the computer is approximately equal to
the decimal value 0.100000000000000005551115123125. Rounding that to 17