switch the core ninja parser to use re2c for the lexer

- Delete the old "Tokenizer" code.
- Write separate tests for the lexer distinct from the parser.
- Switch the parser to use the new code.
- New lexer error output has file:line numbers so e.g. Emacs can
  jump your editor to the syntax error.
- The EvalEnv ($-interpolation) code is now part of the lexer as well.

1 files changed, 2 insertions, 2 deletions

diff --git a/src/graph.h b/src/graph.h
index b483c6d..20765a3 100644
--- a/src/graph.h
+++ b/src/graph.h
@@ -109,10 +109,10 @@ struct Rule {
   const EvalString& description() const { return description_; }
   const EvalString& depfile() const { return depfile_; }
 
- private:
+  // TODO: private:
+
   // Allow the parsers to reach into this object and fill out its fields.
   friend class ManifestParser;
-  friend class ParserTest;
 
   string name_;