bpo-36876: Fix the C analyzer tool. (GH-22841)

The original tool wasn't working right and it was simpler to create a new one, partially re-using some of the old code. At this point the tool runs properly on the master. (Try: ./python Tools/c-analyzer/c-analyzer.py analyze.)  It take ~40 seconds on my machine to analyze the full CPython code base.

Note that we'll need to iron out some OS-specific stuff (e.g. preprocessor). We're okay though since this tool isn't used yet in our workflow. We will also need to verify the analysis results in detail before activating the check in CI, though I'm pretty sure it's close.

https://bugs.python.org/issue36876

1 files changed, 42 insertions, 0 deletions

diff --git a/Tools/c-analyzer/c_common/strutil.py b/Tools/c-analyzer/c_common/strutil.py
new file mode 100644
index 0000000..e7535d4
--- /dev/null
+++ b/Tools/c-analyzer/c_common/strutil.py
@@ -0,0 +1,42 @@
+import logging
+
+
+logger = logging.getLogger(__name__)
+
+
+def unrepr(value):
+    raise NotImplementedError
+
+
+def parse_entries(entries, *, ignoresep=None):
+    for entry in entries:
+        if ignoresep and ignoresep in entry:
+            subentries = [entry]
+        else:
+            subentries = entry.strip().replace(',', ' ').split()
+        for item in subentries:
+            if item.startswith('+'):
+                filename = item[1:]
+                try:
+                    infile = open(filename)
+                except FileNotFoundError:
+                    logger.debug(f'ignored in parse_entries(): +{filename}')
+                    return
+                with infile:
+                    # We read the entire file here to ensure the file
+                    # gets closed sooner rather than later.  Note that
+                    # the file would stay open if this iterator is never
+                    # exchausted.
+                    lines = infile.read().splitlines()
+                for line in _iter_significant_lines(lines):
+                    yield line, filename
+            else:
+                yield item, None
+
+
+def _iter_significant_lines(lines):
+    for line in lines:
+        line = line.partition('#')[0]
+        if not line.strip():
+            continue
+        yield line