Be more careful about file reading

If SCons reads a file to interpret the contents, codecs are a concern.
The File node class has a get_text_contents() method which makes a best
effort at decoding bytes data, but there are other places that don't get
their file contents via that method, and so should do their own careful
decoding - but don't, they just read as text and hope it's okay.

Move the decode-bytes portion out of File.get_text_contents() to
SCons.Util.to_Text() so that everyone that needs this can call it.
Add a couple of additional known BOM codes (after consulting Python's
codecs module).

Note that while get_text_contents acts on nodes, the new (moved) routine
to_Text acts on passed bytes, so it can be used in a non-Node context
as well - for example the Java tool initializer reads a file and tries
to decode it, and can get it wrong (see #3569), this change provides it
some help.

Fixes #3569
FIxes #4462

Signed-off-by: Mats Wichmann <mats@linux.com>

1 files changed, 5 insertions, 3 deletions

diff --git a/SCons/cpp.py b/SCons/cpp.py
index 97aba8c..1093ae2 100644
--- a/SCons/cpp.py
+++ b/SCons/cpp.py
@@ -26,6 +26,8 @@
 import os
 import re
 
+import SCons.Util
+
 # First "subsystem" of regular expressions that we set up:
 #
 # Stuff to turn the C preprocessor directives in a file's contents into
@@ -401,9 +403,9 @@ class PreProcessor:
                 return f
         return None
 
-    def read_file(self, file):
-        with open(file) as f:
-            return f.read()
+    def read_file(self, file) -> str:
+        with open(file, 'rb') as f:
+            return SCons.Util.to_Text(f.read())
 
     # Start and stop processing include lines.