Issue #17911: traceback module overhaul

Provide a way to seed the linecache for a PEP-302 module without actually
loading the code.

Provide a new object API for traceback, including the ability to not lookup
lines at all until the traceback is actually rendered, without any trace of the
original objects being kept alive.

7 files changed, 794 insertions, 157 deletions

diff --git a/Doc/library/linecache.rst b/Doc/library/linecache.rst
index dacf8aa..12d0113 100644
--- a/Doc/library/linecache.rst
+++ b/Doc/library/linecache.rst
@@ -43,6 +43,14 @@ The :mod:`linecache` module defines the following functions:
    changed on disk, and you require the updated version.  If *filename* is omitted,
    it will check all the entries in the cache.
 
+.. function:: lazycache(filename, module_globals)
+
+   Capture enough detail about a non-file based module to permit getting its
+   lines later via :func:`getline` even if *module_globals* is None in the later
+   call. This avoids doing I/O until a line is actually needed, without having
+   to carry the module globals around indefinitely.
+
+   .. versionadded:: 3.5
 
 Example::
 
diff --git a/Doc/library/traceback.rst b/Doc/library/traceback.rst
index 15fbedc..585025d 100644
--- a/Doc/library/traceback.rst
+++ b/Doc/library/traceback.rst
@@ -136,6 +136,130 @@ The module defines the following functions:
 
    .. versionadded:: 3.4
 
+.. function:: walk_stack(f)
+
+   Walk a stack following f.f_back from the given frame, yielding the frame and
+   line number for each frame. If f is None, the current stack is used.
+   This helper is used with *Stack.extract*.
+
+   .. versionadded:: 3.5
+
+.. function:: walk_tb(tb)
+
+   Walk a traceback following tb_next yielding the frame and line number for
+   each frame. This helper is used with *Stack.extract*.
+
+   .. versionadded:: 3.5
+
+The module also defines the following classes:
+
+:class:`TracebackException` Objects
+-----------------------------------
+
+:class:`.TracebackException` objects are created from actual exceptions to
+capture data for later printing in a lightweight fashion.
+
+.. class:: TracebackException(exc_type, exc_value, exc_traceback, limit=None, lookup_lines=True)
+
+   Capture an exception for later rendering. limit, lookup_lines are as for
+   the :class:`.StackSummary` class.
+
+   .. versionadded:: 3.5
+
+.. classmethod:: `.from_exception`(exc, limit=None, lookup_lines=True)
+
+   Capture an exception for later rendering. limit and lookup_lines
+   are as for the :class:`.StackSummary` class.
+
+   .. versionadded:: 3.5
+
+.. attribute:: `.__cause__` A TracebackException of the original *__cause__*.
+
+.. attribute:: `.__context__` A TracebackException of the original *__context__*.
+.. attribute:: `.__suppress_context__` The *__suppress_context__* value from the
+               original exception.
+.. attribute:: `.stack` A `StackSummary` representing the traceback.
+.. attribute:: `.exc_type` The class of the original traceback.
+.. attribute:: `.filename` For syntax errors - the filename where the error
+               occured.
+.. attribute:: `.lineno` For syntax errors - the linenumber where the error
+               occured.
+.. attribute:: `.text` For syntax errors - the text where the error
+               occured.
+.. attribute:: `.offset` For syntax errors - the offset into the text where the
+               error occured.
+.. attribute:: `.msg` For syntax errors - the compiler error message.
+
+.. method::  TracebackException.format(chain=True)
+
+    Format the exception.
+
+    If chain is not *True*, *__cause__* and *__context__* will not be formatted.
+
+    The return value is a generator of strings, each ending in a newline and
+    some containing internal newlines. `print_exception` is a wrapper around
+    this method which just prints the lines to a file.
+
+    The message indicating which exception occurred is always the last
+    string in the output.
+
+   .. versionadded:: 3.5
+
+.. method::  TracebackException.format_exception_only()
+
+    Format the exception part of the traceback.
+
+    The return value is a generator of strings, each ending in a newline.
+
+    Normally, the generator emits a single string; however, for
+    SyntaxError exceptions, it emites several lines that (when
+    printed) display detailed information about where the syntax
+    error occurred.
+
+    The message indicating which exception occurred is always the last
+    string in the output.
+
+   .. versionadded:: 3.5
+
+
+:class:`StackSummary` Objects
+-----------------------------
+
+:class:`.StackSummary` objects represent a call stack ready for formatting.
+
+.. classmethod:: StackSummary.extract(frame_gen, limit=None, lookup_lines=True)
+
+   Construct a StackSummary object from a frame generator (such as is returned by
+   `walk_stack` or `walk_tb`.
+
+   If limit is supplied, only this many frames are taken from frame_gen.
+   If lookup_lines is False, the returned FrameSummary objects will not have read
+   their lines in yet, making the cost of creating the StackSummary cheaper (which
+   may be valuable if it may not actually get formatted).
+
+   .. versionadded:: 3.5
+
+.. classmethod:: StackSummary.from_list(a_list)
+
+   Construct a StackSummary object from a supplied old-style list of tuples. Each
+   tuple should be a 4-tuple with filename, lineno, name, line as the elements.
+
+   .. versionadded:: 3.5
+
+:class:`FrameSummary` Objects
+-----------------------------
+
+FrameSummary objects represent a single frame in a traceback.
+
+.. class:: FrameSummary(filename, lineno, name, lookup_line=True, locals=None, line=None)
+   :noindex:
+
+   Represent a single frame in the traceback or stack that is being formatted
+   or printed. It may optionally have a stringified version of the frames
+   locals included in it. If *lookup_line* is False, the source code is not
+   looked up until the FrameSummary has the :attr:`line` attribute accessed (which
+   also happens when casting it to a tuple). Line may be directly provided, and
+   will prevent line lookups happening at all.
 
 .. _traceback-example:
 
diff --git a/Lib/linecache.py b/Lib/linecache.py
index 02a9eb5..33b0af7 100644
--- a/Lib/linecache.py
+++ b/Lib/linecache.py
@@ -5,6 +5,7 @@ is not found, it will look down the module search path for a file by
 that name.
 """
 
+import functools
 import sys
 import os
 import tokenize
@@ -21,7 +22,9 @@ def getline(filename, lineno, module_globals=None):
 
 # The cache
 
-cache = {} # The cache
+# The cache. Maps filenames to either a thunk which will provide source code,
+# or a tuple (size, mtime, lines, fullname) once loaded.
+cache = {}
 
 
 def clearcache():
@@ -36,6 +39,9 @@ def getlines(filename, module_globals=None):
     Update the cache if it doesn't contain an entry for this file already."""
 
     if filename in cache:
+        entry = cache[filename]
+        if len(entry) == 1:
+            return updatecache(filename, module_globals)
         return cache[filename][2]
     else:
         return updatecache(filename, module_globals)
@@ -54,7 +60,11 @@ def checkcache(filename=None):
             return
 
     for filename in filenames:
-        size, mtime, lines, fullname = cache[filename]
+        entry = cache[filename]
+        if len(entry) == 1:
+            # lazy cache entry, leave it lazy.
+            continue
+        size, mtime, lines, fullname = entry
         if mtime is None:
             continue   # no-op for files loaded via a __loader__
         try:
@@ -72,7 +82,8 @@ def updatecache(filename, module_globals=None):
     and return an empty list."""
 
     if filename in cache:
-        del cache[filename]
+        if len(cache[filename]) != 1:
+            del cache[filename]
     if not filename or (filename.startswith('<') and filename.endswith('>')):
         return []
 
@@ -82,27 +93,23 @@ def updatecache(filename, module_globals=None):
     except OSError:
         basename = filename
 
-        # Try for a __loader__, if available
-        if module_globals and '__loader__' in module_globals:
-            name = module_globals.get('__name__')
-            loader = module_globals['__loader__']
-            get_source = getattr(loader, 'get_source', None)
-
-            if name and get_source:
-                try:
-                    data = get_source(name)
-                except (ImportError, OSError):
-                    pass
-                else:
-                    if data is None:
-                        # No luck, the PEP302 loader cannot find the source
-                        # for this module.
-                        return []
-                    cache[filename] = (
-                        len(data), None,
-                        [line+'\n' for line in data.splitlines()], fullname
-                    )
-                    return cache[filename][2]
+        # Realise a lazy loader based lookup if there is one
+        # otherwise try to lookup right now.
+        if lazycache(filename, module_globals):
+            try:
+                data = cache[filename][0]()
+            except (ImportError, OSError):
+                pass
+            else:
+                if data is None:
+                    # No luck, the PEP302 loader cannot find the source
+                    # for this module.
+                    return []
+                cache[filename] = (
+                    len(data), None,
+                    [line+'\n' for line in data.splitlines()], fullname
+                )
+                return cache[filename][2]
 
         # Try looking through the module search path, which is only useful
         # when handling a relative filename.
@@ -132,3 +139,36 @@ def updatecache(filename, module_globals=None):
     size, mtime = stat.st_size, stat.st_mtime
     cache[filename] = size, mtime, lines, fullname
     return lines
+
+
+def lazycache(filename, module_globals):
+    """Seed the cache for filename with module_globals.
+
+    The module loader will be asked for the source only when getlines is
+    called, not immediately.
+
+    If there is an entry in the cache already, it is not altered.
+
+    :return: True if a lazy load is registered in the cache,
+        otherwise False. To register such a load a module loader with a
+        get_source method must be found, the filename must be a cachable
+        filename, and the filename must not be already cached.
+    """
+    if filename in cache:
+        if len(cache[filename]) == 1:
+            return True
+        else:
+            return False
+    if not filename or (filename.startswith('<') and filename.endswith('>')):
+        return False
+    # Try for a __loader__, if available
+    if module_globals and '__loader__' in module_globals:
+        name = module_globals.get('__name__')
+        loader = module_globals['__loader__']
+        get_source = getattr(loader, 'get_source', None)
+
+        if name and get_source:
+            get_lines = functools.partial(get_source, name)
+            cache[filename] = (get_lines,)
+            return True
+    return False
diff --git a/Lib/test/test_linecache.py b/Lib/test/test_linecache.py
index 5fe0554..2fb8662 100644
--- a/Lib/test/test_linecache.py
+++ b/Lib/test/test_linecache.py
@@ -7,6 +7,7 @@ from test import support
 
 
 FILENAME = linecache.__file__
+NONEXISTENT_FILENAME = FILENAME + '.missing'
 INVALID_NAME = '!@$)(!@#_1'
 EMPTY = ''
 TESTS = 'inspect_fodder inspect_fodder2 mapping_tests'
@@ -126,6 +127,49 @@ class LineCacheTests(unittest.TestCase):
                 self.assertEqual(line, getline(source_name, index + 1))
                 source_list.append(line)
 
+    def test_lazycache_no_globals(self):
+        lines = linecache.getlines(FILENAME)
+        linecache.clearcache()
+        self.assertEqual(False, linecache.lazycache(FILENAME, None))
+        self.assertEqual(lines, linecache.getlines(FILENAME))
+
+    def test_lazycache_smoke(self):
+        lines = linecache.getlines(NONEXISTENT_FILENAME, globals())
+        linecache.clearcache()
+        self.assertEqual(
+            True, linecache.lazycache(NONEXISTENT_FILENAME, globals()))
+        self.assertEqual(1, len(linecache.cache[NONEXISTENT_FILENAME]))
+        # Note here that we're looking up a non existant filename with no
+        # globals: this would error if the lazy value wasn't resolved.
+        self.assertEqual(lines, linecache.getlines(NONEXISTENT_FILENAME))
+
+    def test_lazycache_provide_after_failed_lookup(self):
+        linecache.clearcache()
+        lines = linecache.getlines(NONEXISTENT_FILENAME, globals())
+        linecache.clearcache()
+        linecache.getlines(NONEXISTENT_FILENAME)
+        linecache.lazycache(NONEXISTENT_FILENAME, globals())
+        self.assertEqual(lines, linecache.updatecache(NONEXISTENT_FILENAME))
+
+    def test_lazycache_check(self):
+        linecache.clearcache()
+        linecache.lazycache(NONEXISTENT_FILENAME, globals())
+        linecache.checkcache()
+
+    def test_lazycache_bad_filename(self):
+        linecache.clearcache()
+        self.assertEqual(False, linecache.lazycache('', globals()))
+        self.assertEqual(False, linecache.lazycache('<foo>', globals()))
+
+    def test_lazycache_already_cached(self):
+        linecache.clearcache()
+        lines = linecache.getlines(NONEXISTENT_FILENAME, globals())
+        self.assertEqual(
+            False,
+            linecache.lazycache(NONEXISTENT_FILENAME, globals()))
+        self.assertEqual(4, len(linecache.cache[NONEXISTENT_FILENAME]))
+
+
 def test_main():
     support.run_unittest(LineCacheTests)
 
diff --git a/Lib/test/test_traceback.py b/Lib/test/test_traceback.py
index 6bd6fa6..9ff7548 100644
--- a/Lib/test/test_traceback.py
+++ b/Lib/test/test_traceback.py
@@ -1,6 +1,8 @@
 """Test cases for traceback module"""
 
+from collections import namedtuple
 from io import StringIO
+import linecache
 import sys
 import unittest
 import re
@@ -12,6 +14,11 @@ import textwrap
 import traceback
 
 
+test_code = namedtuple('code', ['co_filename', 'co_name'])
+test_frame = namedtuple('frame', ['f_code', 'f_globals'])
+test_tb = namedtuple('tb', ['tb_frame', 'tb_lineno', 'tb_next'])
+
+
 class SyntaxTracebackCases(unittest.TestCase):
     # For now, a very minimal set of tests.  I want to be sure that
     # formatting of SyntaxErrors works based on changes for 2.1.
@@ -477,6 +484,195 @@ class MiscTracebackCases(unittest.TestCase):
         self.assertEqual(len(inner_frame.f_locals), 0)
 
 
+class TestFrame(unittest.TestCase):
+
+    def test_basics(self):
+        linecache.clearcache()
+        linecache.lazycache("f", globals())
+        f = traceback.FrameSummary("f", 1, "dummy")
+        self.assertEqual(
+            ("f", 1, "dummy", '"""Test cases for traceback module"""'),
+            tuple(f))
+        self.assertEqual(None, f.locals)
+
+    def test_lazy_lines(self):
+        linecache.clearcache()
+        f = traceback.FrameSummary("f", 1, "dummy", lookup_line=False)
+        self.assertEqual(None, f._line)
+        linecache.lazycache("f", globals())
+        self.assertEqual(
+            '"""Test cases for traceback module"""',
+            f.line)
+
+    def test_explicit_line(self):
+        f = traceback.FrameSummary("f", 1, "dummy", line="line")
+        self.assertEqual("line", f.line)
+
+
+class TestStack(unittest.TestCase):
+
+    def test_walk_stack(self):
+        s = list(traceback.walk_stack(None))
+        self.assertGreater(len(s), 10)
+
+    def test_walk_tb(self):
+        try:
+            1/0
+        except Exception:
+            _, _, tb = sys.exc_info()
+        s = list(traceback.walk_tb(tb))
+        self.assertEqual(len(s), 1)
+
+    def test_extract_stack(self):
+        s = traceback.StackSummary.extract(traceback.walk_stack(None))
+        self.assertIsInstance(s, traceback.StackSummary)
+
+    def test_extract_stack_limit(self):
+        s = traceback.StackSummary.extract(traceback.walk_stack(None), limit=5)
+        self.assertEqual(len(s), 5)
+
+    def test_extract_stack_lookup_lines(self):
+        linecache.clearcache()
+        linecache.updatecache('/foo.py', globals())
+        c = test_code('/foo.py', 'method')
+        f = test_frame(c, None)
+        s = traceback.StackSummary.extract(iter([(f, 6)]), lookup_lines=True)
+        linecache.clearcache()
+        self.assertEqual(s[0].line, "import sys")
+
+    def test_extract_stackup_deferred_lookup_lines(self):
+        linecache.clearcache()
+        c = test_code('/foo.py', 'method')
+        f = test_frame(c, None)
+        s = traceback.StackSummary.extract(iter([(f, 6)]), lookup_lines=False)
+        self.assertEqual({}, linecache.cache)
+        linecache.updatecache('/foo.py', globals())
+        self.assertEqual(s[0].line, "import sys")
+
+    def test_from_list(self):
+        s = traceback.StackSummary([('foo.py', 1, 'fred', 'line')])
+        self.assertEqual(
+            ['  File "foo.py", line 1, in fred\n    line\n'],
+            s.format())
+
+    def test_format_smoke(self):
+        # For detailed tests see the format_list tests, which consume the same
+        # code.
+        s = traceback.StackSummary([('foo.py', 1, 'fred', 'line')])
+        self.assertEqual(
+            ['  File "foo.py", line 1, in fred\n    line\n'],
+            s.format())
+
+
+
+class TestTracebackException(unittest.TestCase):
+
+    def test_smoke(self):
+        try:
+            1/0
+        except Exception:
+            exc_info = sys.exc_info()
+            exc = traceback.TracebackException(*exc_info)
+            expected_stack = traceback.StackSummary.extract(
+                traceback.walk_tb(exc_info[2]))
+        self.assertEqual(None, exc.__cause__)
+        self.assertEqual(None, exc.__context__)
+        self.assertEqual(False, exc.__suppress_context__)
+        self.assertEqual(expected_stack, exc.stack)
+        self.assertEqual(exc_info[0], exc.exc_type)
+        self.assertEqual(str(exc_info[1]), str(exc))
+
+    def test_from_exception(self):
+        # Check all the parameters are accepted.
+        def foo():
+            1/0
+        try:
+            foo()
+        except Exception as e:
+            exc_info = sys.exc_info()
+            self.expected_stack = traceback.StackSummary.extract(
+                traceback.walk_tb(exc_info[2]), limit=1, lookup_lines=False,
+                capture_locals=True)
+            self.exc = traceback.TracebackException.from_exception(
+                e, limit=1, lookup_lines=False, capture_locals=True)
+        expected_stack = self.expected_stack
+        exc = self.exc
+        self.assertEqual(None, exc.__cause__)
+        self.assertEqual(None, exc.__context__)
+        self.assertEqual(False, exc.__suppress_context__)
+        self.assertEqual(expected_stack, exc.stack)
+        self.assertEqual(exc_info[0], exc.exc_type)
+        self.assertEqual(str(exc_info[1]), str(exc))
+
+    def test_cause(self):
+        try:
+            try:
+                1/0
+            finally:
+                exc_info_context = sys.exc_info()
+                exc_context = traceback.TracebackException(*exc_info_context)
+                cause = Exception("cause")
+                raise Exception("uh oh") from cause
+        except Exception:
+            exc_info = sys.exc_info()
+            exc = traceback.TracebackException(*exc_info)
+            expected_stack = traceback.StackSummary.extract(
+                traceback.walk_tb(exc_info[2]))
+        exc_cause = traceback.TracebackException(Exception, cause, None)
+        self.assertEqual(exc_cause, exc.__cause__)
+        self.assertEqual(exc_context, exc.__context__)
+        self.assertEqual(True, exc.__suppress_context__)
+        self.assertEqual(expected_stack, exc.stack)
+        self.assertEqual(exc_info[0], exc.exc_type)
+        self.assertEqual(str(exc_info[1]), str(exc))
+
+    def test_context(self):
+        try:
+            try:
+                1/0
+            finally:
+                exc_info_context = sys.exc_info()
+                exc_context = traceback.TracebackException(*exc_info_context)
+                raise Exception("uh oh")
+        except Exception:
+            exc_info = sys.exc_info()
+            exc = traceback.TracebackException(*exc_info)
+            expected_stack = traceback.StackSummary.extract(
+                traceback.walk_tb(exc_info[2]))
+        self.assertEqual(None, exc.__cause__)
+        self.assertEqual(exc_context, exc.__context__)
+        self.assertEqual(False, exc.__suppress_context__)
+        self.assertEqual(expected_stack, exc.stack)
+        self.assertEqual(exc_info[0], exc.exc_type)
+        self.assertEqual(str(exc_info[1]), str(exc))
+
+    def test_limit(self):
+        def recurse(n):
+            if n:
+                recurse(n-1)
+            else:
+                1/0
+        try:
+            recurse(10)
+        except Exception:
+            exc_info = sys.exc_info()
+            exc = traceback.TracebackException(*exc_info, limit=5)
+            expected_stack = traceback.StackSummary.extract(
+                traceback.walk_tb(exc_info[2]), limit=5)
+        self.assertEqual(expected_stack, exc.stack)
+
+    def test_lookup_lines(self):
+        linecache.clearcache()
+        e = Exception("uh oh")
+        c = test_code('/foo.py', 'method')
+        f = test_frame(c, None)
+        tb = test_tb(f, 6, None)
+        exc = traceback.TracebackException(Exception, e, tb, lookup_lines=False)
+        self.assertEqual({}, linecache.cache)
+        linecache.updatecache('/foo.py', globals())
+        self.assertEqual(exc.stack[0].line, "import sys")
+
+
 def test_main():
     run_unittest(__name__)
 
diff --git a/Lib/traceback.py b/Lib/traceback.py
index c1ab36e..22544ac 100644
--- a/Lib/traceback.py
+++ b/Lib/traceback.py
@@ -14,19 +14,12 @@ __all__ = ['extract_stack', 'extract_tb', 'format_exception',
 # Formatting and printing lists of traceback lines.
 #
 
-def _format_list_iter(extracted_list):
-    for filename, lineno, name, line in extracted_list:
-        item = '  File "{}", line {}, in {}\n'.format(filename, lineno, name)
-        if line:
-            item = item + '    {}\n'.format(line.strip())
-        yield item
-
 def print_list(extracted_list, file=None):
     """Print the list of tuples as returned by extract_tb() or
     extract_stack() as a formatted stack trace to the given file."""
     if file is None:
         file = sys.stderr
-    for item in _format_list_iter(extracted_list):
+    for item in StackSummary.from_list(extracted_list).format():
         print(item, file=file, end="")
 
 def format_list(extracted_list):
@@ -39,45 +32,12 @@ def format_list(extracted_list):
     the strings may contain internal newlines as well, for those items
     whose source text line is not None.
     """
-    return list(_format_list_iter(extracted_list))
+    return StackSummary.from_list(extracted_list).format()
 
 #
 # Printing and Extracting Tracebacks.
 #
 
-# extractor takes curr and needs to return a tuple of:
-# - Frame object
-# - Line number
-# - Next item (same type as curr)
-# In practice, curr is either a traceback or a frame.
-def _extract_tb_or_stack_iter(curr, limit, extractor):
-    if limit is None:
-        limit = getattr(sys, 'tracebacklimit', None)
-
-    n = 0
-    while curr is not None and (limit is None or n < limit):
-        f, lineno, next_item = extractor(curr)
-        co = f.f_code
-        filename = co.co_filename
-        name = co.co_name
-
-        linecache.checkcache(filename)
-        line = linecache.getline(filename, lineno, f.f_globals)
-
-        if line:
-            line = line.strip()
-        else:
-            line = None
-
-        yield (filename, lineno, name, line)
-        curr = next_item
-        n += 1
-
-def _extract_tb_iter(tb, limit):
-    return _extract_tb_or_stack_iter(
-                tb, limit,
-                operator.attrgetter("tb_frame", "tb_lineno", "tb_next"))
-
 def print_tb(tb, limit=None, file=None):
     """Print up to 'limit' stack trace entries from the traceback 'tb'.
 
@@ -90,7 +50,7 @@ def print_tb(tb, limit=None, file=None):
 
 def format_tb(tb, limit=None):
     """A shorthand for 'format_list(extract_tb(tb, limit))'."""
-    return format_list(extract_tb(tb, limit=limit))
+    return extract_tb(tb, limit=limit).format()
 
 def extract_tb(tb, limit=None):
     """Return list of up to limit pre-processed entries from traceback.
@@ -103,7 +63,7 @@ def extract_tb(tb, limit=None):
     leading and trailing whitespace stripped; if the source is not
     available it is None.
     """
-    return list(_extract_tb_iter(tb, limit=limit))
+    return StackSummary.extract(walk_tb(tb), limit=limit)
 
 #
 # Exception formatting and output.
@@ -111,47 +71,12 @@ def extract_tb(tb, limit=None):
 
 _cause_message = (
     "\nThe above exception was the direct cause "
-    "of the following exception:\n")
+    "of the following exception:\n\n")
 
 _context_message = (
     "\nDuring handling of the above exception, "
-    "another exception occurred:\n")
-
-def _iter_chain(exc, custom_tb=None, seen=None):
-    if seen is None:
-        seen = set()
-    seen.add(exc)
-    its = []
-    context = exc.__context__
-    cause = exc.__cause__
-    if cause is not None and cause not in seen:
-        its.append(_iter_chain(cause, False, seen))
-        its.append([(_cause_message, None)])
-    elif (context is not None and
-          not exc.__suppress_context__ and
-          context not in seen):
-        its.append(_iter_chain(context, None, seen))
-        its.append([(_context_message, None)])
-    its.append([(exc, custom_tb or exc.__traceback__)])
-    # itertools.chain is in an extension module and may be unavailable
-    for it in its:
-        yield from it
-
-def _format_exception_iter(etype, value, tb, limit, chain):
-    if chain:
-        values = _iter_chain(value, tb)
-    else:
-        values = [(value, tb)]
-
-    for value, tb in values:
-        if isinstance(value, str):
-            # This is a cause/context message line
-            yield value + '\n'
-            continue
-        if tb:
-            yield 'Traceback (most recent call last):\n'
-            yield from _format_list_iter(_extract_tb_iter(tb, limit=limit))
-        yield from _format_exception_only_iter(type(value), value)
+    "another exception occurred:\n\n")
+
 
 def print_exception(etype, value, tb, limit=None, file=None, chain=True):
     """Print exception up to 'limit' stack trace entries from 'tb' to 'file'.
@@ -166,9 +91,11 @@ def print_exception(etype, value, tb, limit=None, file=None, chain=True):
     """
     if file is None:
         file = sys.stderr
-    for line in _format_exception_iter(etype, value, tb, limit, chain):
+    for line in TracebackException(
+            etype, value, tb, limit=limit).format(chain=chain):
         print(line, file=file, end="")
 
+
 def format_exception(etype, value, tb, limit=None, chain=True):
     """Format a stack trace and the exception information.
 
@@ -178,7 +105,9 @@ def format_exception(etype, value, tb, limit=None, chain=True):
     these lines are concatenated and printed, exactly the same text is
     printed as does print_exception().
     """
-    return list(_format_exception_iter(etype, value, tb, limit, chain))
+    return list(TracebackException(
+        etype, value, tb, limit=limit).format(chain=chain))
+
 
 def format_exception_only(etype, value):
     """Format the exception part of a traceback.
@@ -196,46 +125,14 @@ def format_exception_only(etype, value):
     string in the list.
 
     """
-    return list(_format_exception_only_iter(etype, value))
-
-def _format_exception_only_iter(etype, value):
-    # Gracefully handle (the way Python 2.4 and earlier did) the case of
-    # being called with (None, None).
-    if etype is None:
-        yield _format_final_exc_line(etype, value)
-        return
-
-    stype = etype.__qualname__
-    smod = etype.__module__
-    if smod not in ("__main__", "builtins"):
-        stype = smod + '.' + stype
-
-    if not issubclass(etype, SyntaxError):
-        yield _format_final_exc_line(stype, value)
-        return
-
-    # It was a syntax error; show exactly where the problem was found.
-    filename = value.filename or "<string>"
-    lineno = str(value.lineno) or '?'
-    yield '  File "{}", line {}\n'.format(filename, lineno)
-
-    badline = value.text
-    offset = value.offset
-    if badline is not None:
-        yield '    {}\n'.format(badline.strip())
-        if offset is not None:
-            caretspace = badline.rstrip('\n')
-            offset = min(len(caretspace), offset) - 1
-            caretspace = caretspace[:offset].lstrip()
-            # non-space whitespace (likes tabs) must be kept for alignment
-            caretspace = ((c.isspace() and c or ' ') for c in caretspace)
-            yield '    {}^\n'.format(''.join(caretspace))
-    msg = value.msg or "<no detail available>"
-    yield "{}: {}\n".format(stype, msg)
+    return list(TracebackException(etype, value, None).format_exception_only())
+
+
+# -- not offical API but folk probably use these two functions.
 
 def _format_final_exc_line(etype, value):
     valuestr = _some_str(value)
-    if value is None or not valuestr:
+    if value == 'None' or value is None or not valuestr:
         line = "%s\n" % etype
     else:
         line = "%s: %s\n" % (etype, valuestr)
@@ -247,6 +144,8 @@ def _some_str(value):
     except:
         return '<unprintable %s object>' % type(value).__name__
 
+# --
+
 def print_exc(limit=None, file=None, chain=True):
     """Shorthand for 'print_exception(*sys.exc_info(), limit, file)'."""
     print_exception(*sys.exc_info(), limit=limit, file=file, chain=chain)
@@ -267,15 +166,6 @@ def print_last(limit=None, file=None, chain=True):
 # Printing and Extracting Stacks.
 #
 
-def _extract_stack_iter(f, limit=None):
-    return _extract_tb_or_stack_iter(
-                f, limit, lambda f: (f, f.f_lineno, f.f_back))
-
-def _get_stack(f):
-    if f is None:
-        f = sys._getframe().f_back.f_back
-    return f
-
 def print_stack(f=None, limit=None, file=None):
     """Print a stack trace from its invocation point.
 
@@ -283,11 +173,13 @@ def print_stack(f=None, limit=None, file=None):
     stack frame at which to start. The optional 'limit' and 'file'
     arguments have the same meaning as for print_exception().
     """
-    print_list(extract_stack(_get_stack(f), limit=limit), file=file)
+    print_list(extract_stack(f, limit=limit), file=file)
+
 
 def format_stack(f=None, limit=None):
     """Shorthand for 'format_list(extract_stack(f, limit))'."""
-    return format_list(extract_stack(_get_stack(f), limit=limit))
+    return format_list(extract_stack(f, limit=limit))
+
 
 def extract_stack(f=None, limit=None):
     """Extract the raw traceback from the current stack frame.
@@ -298,10 +190,11 @@ def extract_stack(f=None, limit=None):
     line number, function name, text), and the entries are in order
     from oldest to newest stack frame.
     """
-    stack = list(_extract_stack_iter(_get_stack(f), limit=limit))
+    stack = StackSummary.extract(walk_stack(f), limit=limit)
     stack.reverse()
     return stack
 
+
 def clear_frames(tb):
     "Clear all references to local variables in the frames of a traceback."
     while tb is not None:
@@ -311,3 +204,328 @@ def clear_frames(tb):
             # Ignore the exception raised if the frame is still executing.
             pass
         tb = tb.tb_next
+
+
+class FrameSummary:
+    """A single frame from a traceback.
+
+    - :attr:`filename` The filename for the frame.
+    - :attr:`lineno` The line within filename for the frame that was
+      active when the frame was captured.
+    - :attr:`name` The name of the function or method that was executing
+      when the frame was captured.
+    - :attr:`line` The text from the linecache module for the
+      of code that was running when the frame was captured.
+    - :attr:`locals` Either None if locals were not supplied, or a dict
+      mapping the name to the str() of the variable.
+    """
+
+    __slots__ = ('filename', 'lineno', 'name', '_line', 'locals')
+
+    def __init__(self, filename, lineno, name, lookup_line=True, locals=None,
+            line=None):
+        """Construct a FrameSummary.
+
+        :param lookup_line: If True, `linecache` is consulted for the source
+            code line. Otherwise, the line will be looked up when first needed.
+        :param locals: If supplied the frame locals, which will be captured as
+            strings.
+        :param line: If provided, use this instead of looking up the line in
+            the linecache.
+        """
+        self.filename = filename
+        self.lineno = lineno
+        self.name = name
+        self._line = line
+        if lookup_line:
+            self.line
+        self.locals = \
+            dict((k, str(v)) for k, v in locals.items()) if locals else None
+
+    def __eq__(self, other):
+        return (self.filename == other.filename and
+                self.lineno == other.lineno and
+                self.name == other.name and
+                self.locals == other.locals)
+
+    def __getitem__(self, pos):
+        return (self.filename, self.lineno, self.name, self.line)[pos]
+
+    def __iter__(self):
+        return iter([self.filename, self.lineno, self.name, self.line])
+
+    def __repr__(self):
+        return "<FrameSummary file {filename}, line {lineno} in {name}>".format(
+            filename=self.filename, lineno=self.lineno, name=self.name)
+
+    @property
+    def line(self):
+        if self._line is None:
+            self._line = linecache.getline(self.filename, self.lineno).strip()
+        return self._line
+
+
+def walk_stack(f):
+    """Walk a stack yielding the frame and line number for each frame.
+
+    This will follow f.f_back from the given frame. If no frame is given, the
+    current stack is used. Usually used with StackSummary.extract.
+    """
+    if f is None:
+        f = sys._getframe().f_back.f_back
+    while f is not None:
+        yield f, f.f_lineno
+        f = f.f_back
+
+
+def walk_tb(tb):
+    """Walk a traceback yielding the frame and line number for each frame.
+
+    This will follow tb.tb_next (and thus is in the opposite order to
+    walk_stack). Usually used with StackSummary.extract.
+    """
+    while tb is not None:
+        yield tb.tb_frame, tb.tb_lineno
+        tb = tb.tb_next
+
+
+class StackSummary(list):
+    """A stack of frames."""
+
+    @classmethod
+    def extract(klass, frame_gen, limit=None, lookup_lines=True):
+        """Create a StackSummary from a traceback or stack object.
+
+        :param frame_gen: A generator that yields (frame, lineno) tuples to
+            include in the stack.
+        :param limit: None to include all frames or the number of frames to
+            include.
+        :param lookup_lines: If True, lookup lines for each frame immediately,
+            otherwise lookup is deferred until the frame is rendered.
+        """
+        if limit is None:
+            limit = getattr(sys, 'tracebacklimit', None)
+
+        result = klass()
+        fnames = set()
+        for pos, (f, lineno) in enumerate(frame_gen):
+            if limit is not None and pos >= limit:
+                break
+            co = f.f_code
+            filename = co.co_filename
+            name = co.co_name
+
+            fnames.add(filename)
+            linecache.lazycache(filename, f.f_globals)
+            # Must defer line lookups until we have called checkcache.
+            result.append(FrameSummary(filename, lineno, name, lookup_line=False))
+        for filename in fnames:
+            linecache.checkcache(filename)
+        # If immediate lookup was desired, trigger lookups now.
+        if lookup_lines:
+            for f in result:
+                f.line
+        return result
+
+    @classmethod
+    def from_list(klass, a_list):
+        """Create a StackSummary from a simple list of tuples.
+
+        This method supports the older Python API. Each tuple should be a
+        4-tuple with (filename, lineno, name, line) elements.
+        """
+        if isinstance(a_list, StackSummary):
+            return StackSummary(a_list)
+        result = StackSummary()
+        for filename, lineno, name, line in a_list:
+            result.append(FrameSummary(filename, lineno, name, line=line))
+        return result
+
+    def format(self):
+        """Format the stack ready for printing.
+
+        Returns a list of strings ready for printing.  Each string in the
+        resulting list corresponds to a single frame from the stack.
+        Each string ends in a newline; the strings may contain internal
+        newlines as well, for those items with source text lines.
+        """
+        result = []
+        for filename, lineno, name, line in self:
+            item = '  File "{}", line {}, in {}\n'.format(filename, lineno, name)
+            if line:
+                item = item + '    {}\n'.format(line.strip())
+            result.append(item)
+        return result
+
+
+class TracebackException:
+    """An exception ready for rendering.
+
+    The traceback module captures enough attributes from the original exception
+    to this intermediary form to ensure that no references are held, while
+    still being able to fully print or format it.
+
+    Use `from_exception` to create TracebackException instances from exception
+    objects, or the constructor to create TracebackException instances from
+    individual components.
+
+    - :attr:`__cause__` A TracebackException of the original *__cause__*.
+    - :attr:`__context__` A TracebackException of the original *__context__*.
+    - :attr:`__suppress_context__` The *__suppress_context__* value from the
+      original exception.
+    - :attr:`stack` A `StackSummary` representing the traceback.
+    - :attr:`exc_type` The class of the original traceback.
+    - :attr:`filename` For syntax errors - the filename where the error
+      occured.
+    - :attr:`lineno` For syntax errors - the linenumber where the error
+      occured.
+    - :attr:`text` For syntax errors - the text where the error
+      occured.
+    - :attr:`offset` For syntax errors - the offset into the text where the
+      error occured.
+    - :attr:`msg` For syntax errors - the compiler error message.
+    """
+
+    def __init__(self, exc_type, exc_value, exc_traceback, limit=None,
+            lookup_lines=True, _seen=None):
+        # NB: we need to accept exc_traceback, exc_value, exc_traceback to
+        # permit backwards compat with the existing API, otherwise we
+        # need stub thunk objects just to glue it together.
+        # Handle loops in __cause__ or __context__.
+        if _seen is None:
+            _seen = set()
+        _seen.add(exc_value)
+        # Gracefully handle (the way Python 2.4 and earlier did) the case of
+        # being called with no type or value (None, None, None).
+        if (exc_value and exc_value.__cause__ is not None
+            and exc_value.__cause__ not in _seen):
+            cause = TracebackException(
+                type(exc_value.__cause__),
+                exc_value.__cause__,
+                exc_value.__cause__.__traceback__,
+                limit=limit,
+                lookup_lines=False,
+                _seen=_seen)
+        else:
+            cause = None
+        if (exc_value and exc_value.__context__ is not None
+            and exc_value.__context__ not in _seen):
+            context = TracebackException(
+                type(exc_value.__context__),
+                exc_value.__context__,
+                exc_value.__context__.__traceback__,
+                limit=limit,
+                lookup_lines=False,
+                _seen=_seen)
+        else:
+            context = None
+        self.__cause__ = cause
+        self.__context__ = context
+        self.__suppress_context__ = \
+            exc_value.__suppress_context__ if exc_value else False
+        # TODO: locals.
+        self.stack = StackSummary.extract(
+            walk_tb(exc_traceback), limit=limit, lookup_lines=lookup_lines)
+        self.exc_type = exc_type
+        # Capture now to permit freeing resources: only complication is in the
+        # unofficial API _format_final_exc_line
+        self._str = _some_str(exc_value)
+        if exc_type and issubclass(exc_type, SyntaxError):
+            # Handle SyntaxError's specially
+            self.filename = exc_value.filename
+            self.lineno = str(exc_value.lineno)
+            self.text = exc_value.text
+            self.offset = exc_value.offset
+            self.msg = exc_value.msg
+        if lookup_lines:
+            self._load_lines()
+
+    @classmethod
+    def from_exception(self, exc, *args, **kwargs):
+        """Create a TracebackException from an exception."""
+        return TracebackException(
+            type(exc), exc, exc.__traceback__, *args, **kwargs)
+
+    def _load_lines(self):
+        """Private API. force all lines in the stack to be loaded."""
+        for frame in self.stack:
+            frame.line
+        if self.__context__:
+            self.__context__._load_lines()
+        if self.__cause__:
+            self.__cause__._load_lines()
+
+    def __eq__(self, other):
+        return self.__dict__ == other.__dict__
+
+    def __str__(self):
+        return self._str
+
+    def format_exception_only(self):
+        """Format the exception part of the traceback.
+
+        The return value is a generator of strings, each ending in a newline.
+
+        Normally, the generator emits a single string; however, for
+        SyntaxError exceptions, it emites several lines that (when
+        printed) display detailed information about where the syntax
+        error occurred.
+
+        The message indicating which exception occurred is always the last
+        string in the output.
+        """
+        if self.exc_type is None:
+            yield _format_final_exc_line(None, self._str)
+            return
+
+        stype = self.exc_type.__qualname__
+        smod = self.exc_type.__module__
+        if smod not in ("__main__", "builtins"):
+            stype = smod + '.' + stype
+
+        if not issubclass(self.exc_type, SyntaxError):
+            yield _format_final_exc_line(stype, self._str)
+            return
+
+        # It was a syntax error; show exactly where the problem was found.
+        filename = self.filename or "<string>"
+        lineno = str(self.lineno) or '?'
+        yield '  File "{}", line {}\n'.format(filename, lineno)
+
+        badline = self.text
+        offset = self.offset
+        if badline is not None:
+            yield '    {}\n'.format(badline.strip())
+            if offset is not None:
+                caretspace = badline.rstrip('\n')
+                offset = min(len(caretspace), offset) - 1
+                caretspace = caretspace[:offset].lstrip()
+                # non-space whitespace (likes tabs) must be kept for alignment
+                caretspace = ((c.isspace() and c or ' ') for c in caretspace)
+                yield '    {}^\n'.format(''.join(caretspace))
+        msg = self.msg or "<no detail available>"
+        yield "{}: {}\n".format(stype, msg)
+
+    def format(self, chain=True):
+        """Format the exception.
+
+        If chain is not *True*, *__cause__* and *__context__* will not be formatted.
+
+        The return value is a generator of strings, each ending in a newline and
+        some containing internal newlines. `print_exception` is a wrapper around
+        this method which just prints the lines to a file.
+
+        The message indicating which exception occurred is always the last
+        string in the output.
+        """
+        if chain:
+            if self.__cause__ is not None:
+                yield from self.__cause__.format(chain=chain)
+                yield _cause_message
+            elif (self.__context__ is not None and
+                not self.__suppress_context__):
+                yield from self.__context__.format(chain=chain)
+                yield _context_message
+        yield 'Traceback (most recent call last):\n'
+        yield from self.stack.format()
+        yield from self.format_exception_only()
diff --git a/Misc/NEWS b/Misc/NEWS
index 7f1b7ad..40d8228 100644
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -441,6 +441,13 @@ Library
   now clears its internal reference to the selector mapping to break a
   reference cycle. Initial patch written by Martin Richard.
 
+- Issue #17911: Provide a way to seed the linecache for a PEP-302 module
+  without actually loading the code.
+
+- Issue #17911: Provide a new object API for traceback, including the ability
+  to not lookup lines at all until the traceback is actually rendered, without
+  any trace of the original objects being kept alive.
+
 - Issue #19777: Provide a home() classmethod on Path objects.  Contributed
   by Victor Salgado and Mayank Tripathi.