diff options
Diffstat (limited to 'doc/manual.asciidoc')
| -rw-r--r-- | doc/manual.asciidoc | 506 |
1 files changed, 506 insertions, 0 deletions
diff --git a/doc/manual.asciidoc b/doc/manual.asciidoc new file mode 100644 index 0000000..7aa000e --- /dev/null +++ b/doc/manual.asciidoc @@ -0,0 +1,506 @@ +Ninja +===== +Evan Martin <martine@danga.com> + + +Introduction +------------ + +Ninja is yet another build system. It takes as input the +interdependencies of files (typically source code and output +executables) and orchestrates building them, _quickly_. + +Ninja joins a sea of other build systems. Its distinguishing goal is +to be fast. It is born from my work on the Chromium browser project, +which has over 30,000 source files and whose other build systems +(including one built from custom non-recursive Makefiles) can take ten +seconds to start building after changing one file. Ninja is under a +second. + + +Philosophical overview +~~~~~~~~~~~~~~~~~~~~~~ + +Build systems get slow when they need to make decisions. When you are +in a edit-compile cycle you want it to be as fast as possible -- you +want the build system to do the minimum work necessary to figure out +what needs to be built immediately. + +Ninja contains the barest functionality necessary to describe +arbitrary dependency graphs. Its lack of syntax makes it impossible +to express complex decisions. Instead, Ninja is intended to be used +with a separate program generating its input files. The generator +program (like the `./configure` found in autotools projects) can +analyze system dependencies and make as many decisions as possible up +front so that incremental builds stay fast. Going beyond autotools, +even build-time decisions like "which compiler flags should I use?" +or "should I build a debug or release-mode binary?" belong in the +`.ninja` file generator. + +Conceptual overview +~~~~~~~~~~~~~~~~~~~ + +Ninja evaluates a graph of dependencies between files, and runs +whichever commands are necessary to make your build target up to date. +If you are familiar with Make, Ninja is very similar. + +A build file (default name: `build.ninja`) provides a list of _rules_ +-- short names for longer commands, like how to run the compiler -- +along with a list of _build_ statements saying how to build files +using the rules -- which rule to apply to which inputs to produce +which outputs. + +Conceptually, `build` statements describe the dependency graph of your +project, while `rule` statements describe how to generate the files +along a given edge of the graph. + +Design goals +~~~~~~~~~~~~ + +Here are some of the design goals of Ninja: + +* very fast (i.e., instant) incremental builds, even for very large + projects. + +* very little policy about how code is built; "explicit is better than + implicit". + +* get dependencies correct, and in particular situations that are + difficult to get right with Makefiles (e.g. outputs need an implicit + dependency on the command line used to generate them; to build C + source code you need to use gcc's `-M` flags for header + dependencies). + +* when convenience and speed are in conflict, prefer speed. + +Some explicit _non-goals_: + +* convenient syntax for writing build files by hand. _You should + generate your ninja files using another program_. This is how we + can sidestep many policy decisions. + +* built-in rules. _Out of the box, Ninja has no rules for + e.g. compiling C code._ + +* build-time customization of the build. _Options belong in + the program that generates the ninja files_. + +* build-time decision-making ability such as conditionals or search + paths. _Making decisions is slow._ + +To restate, Ninja is faster than other build systems because it is +painfully simple. You must tell Ninja exactly what to do when you +create your project's `.ninja` files. + +Comparison to GNU make +~~~~~~~~~~~~~~~~~~~~~~ + +Ninja is closest in spirit and functionality to make. But +fundamentally, make has a lot of _features_: suffix rules, functions, +built-in rules that e.g. search for RCS files when building source. +Many projects find make alone adequate for their build problems. In +contrast, Ninja has almost no features; just those necessary to get +builds correct while punting most complexity to generation of the +ninja input files. Ninja by itself is unlikely to be useful for most +projects. + +Here are some of the features ninja adds to make. (These sorts of +features can often be implemented using more complicated Makefiles, +but they are not part of make itself.) + +* A Ninja rule may point at a path for extra implicit dependency + information. This makes it easy to get header dependencies correct + for C/C++ code. + +* A build edge may have multiple outputs. + +* Outputs implicitly depend on the command line that was used to generate + them, which means that changing e.g. compilation flags will cause + the outputs to rebuild. + +* Output directories are always implicitly created before running the + command that relies on them. + +* Rules can provide shorter descriptions of the command being run, so + you can print e.g. `CC foo.o` instead of a long command line while + building. + +* Command output is always buffered. This means commands running in + parallel don't interleave their output, and when a command fails we + can print its failure output next to the full command line that + produced the failure. + + +Getting started +--------------- + +The included `bootstrap.sh` should hopefully produce a working `ninja` +binary, by first blindly compiling all non-test files together then +re-building Ninja using itself. + +Usage is currently just + +---------------- +ninja target +---------------- + +where `target` is a known output described by `build.ninja` in the +current directory. + +There is no installation step; the only files of interest to a user +are the resulting binary and this manual. + + +Creating .ninja files +--------------------- +Here's a basic `.ninja` file that demonstrates most of the syntax. +It will be used as an example for the following sections. + +--------------------------------- +cflags = -Wall + +rule cc + command = gcc $cflags -c $in -o $out + +build foo.o: cc foo.c +--------------------------------- + +Variables +~~~~~~~~~ +Despite the non-goal of being convenient to write by hand, to keep +build files readable (debuggable), Ninja supports declaring shorter +reusable names for strings. A declaration like the following + +---------------- +cflags = -g +---------------- + +can be used on the right side of an equals sign, dereferencing it with +a dollar sign, like this: + +---------------- +rule cc + command = gcc $cflags -c $in -o $out +---------------- + +Variables can also be referenced using curly braces like `${in}`. + +Variables might better be called "bindings", in that a given variable +cannot be changed, only shadowed. There is more on how shadowing works +later in this document. + +Rules +~~~~~ + +Rules declare a short name for a command line. They begin with a line +consisting of the `rule` keyword and a name for the rule. Then +follows an indented set of `variable = value` lines. + +The basic example above declares a new rule named `cc`, along with the +command to run. (In the context of a rule, the `command` variable is +special and defines the command to run. A full list of special +variables is provided in <<ref_rule,the reference>>.) + +Within the context of a rule, two additional special variables are +available: `$in` expands to the list of input files (`foo.c`) and +`$out` to the output file (`foo.o`) for the command. + + +Build statements +~~~~~~~~~~~~~~~~ + +Build statements declare a relationship between input and output +files. They begin with the `build` keyword, and have the format ++build _outputs_: _rulename_ _inputs_+. Such a declaration says that +all of the output files are derived from the input files. When the +output files are missing or when the inputs change, Ninja will run the +rule to regenerate the outputs. + +The basic example above describes how to build `foo.o`, using the `cc` +rule. + +In the scope of a `build` block (including in the evaluation of its +associated `rule`), the variable `$in` is the list of inputs and the +variable `$out` is the list of outputs. + +A build statement may be followed by an indented set of `key = value` +pairs, much like a rule. These variables will shadow any variables +when evaluating the variables in the command. For example: + +---------------- +cflags = -Wall -Werror +rule cc + command = gcc $cflags -c $in -o $out + +# If left unspecified, builds get the outer $cflags. +build foo.o: cc foo.c + +# But you can can shadow variables like cflags for a particular build. +build special.o: cc special.c + cflags = -Wall + +# The variable was only shadowed for the scope of special.o; +# Subsequent build lines get the outer (original) cflags. +build bar.o: cc bar.c + +---------------- + +For more discussion of how scoping works, consult <<ref_scope,the +reference>>. + +If you need more complicated information passed from the build +statement to the rule (for example, if the rule needs "the file +extension of the first input"), pass that through as an extra +variable, like how `cflags` is passed above. + + +The `phony` rule +~~~~~~~~~~~~~~~~ + +The special rule name `phony` can be used to create aliases for other +targets. For example: + +---------------- +build all: phony some/file/in/a/faraway/subdir +---------------- + +This makes `ninja all` build the other files. Semantically, the +`phony` rule is equivalent to a plain rule where the `command` does +nothing, but phony rules are handled specially in that they aren't +printed when run, logged (see below), nor do they contribute to the +command count printed as part of the build process. + + +The Ninja log +~~~~~~~~~~~~~ + +For each built file, Ninja keeps a log of the command used to build +it. Using this log Ninja can know when an existing output was built +with a different command line than the build files specify (i.e., the +command line changed) and knows to rebuild the file. + +The log file is kept in the build root in a file called `.ninja_log`. +If you provide a variable named `builddir` in the outermost scope, +`.ninja_log` will be kept in that directory instead. + + +Generating Ninja files +---------------------- + +A work-in-progress patch to http://gyp.googlecode.com[gyp, the system +used to generate build files for the Chromium browser] to generate +ninja files for Linux is included in the source distribution. + +Conceptually, you could coax Automake into producing ninja files as +well, but I haven't tried it. It may very well be the case that most +projects use too much Makefile syntax in their `.am` files for this to +work. + +Extra tools +----------- + +The `-t` flag on the Ninja command line runs some tools that I have +found useful during Ninja's development. The current tools are: + +`query`:: dump the inputs and outputs of a given target. + +`browse`:: browse the dependency graph in a web browser. Clicking a +file focuses the view on that file, showing inputs and outputs. This +feature requires a Python installation. + +`graph`:: output a file in the syntax used by `graphviz`, a automatic +graph layout tool. Use it like: +ninja -t graph _target_ | dot -Tpng +-ograph.png /dev/stdin+ . In the Ninja source tree, `ninja graph` +generates an image for Ninja itself. If no target is given generate a +graph for all root targets. + +`targets`:: output a list of targets either by rule or by depth. If used +like this +ninja -t targets rule _name_+ it prints the list of targets +using the given rule to be built. If no rule is given, it prints the source +files (the leaves of the graph). If used like this ++ninja -t targets depth _digit_+ it +prints the list of targets in a depth-first manner starting by the root +targets (the ones with no outputs). Indentation is used to mark dependencies. +If the depth is zero it prints all targets. If no arguments are provided ++ninja -t targets depth 1+ is assumed. In this mode targets may be listed +several times. If used like this +ninja -t targets all+ it +prints all the targets available without indentation and it is way faster +than the _depth_ mode. It returns non-zero if an error occurs. + +`rules`:: output the list of all rules with their description if they have +one. It can be used to know which rule name to pass to ++ninja -t targets rule _name_+. + +`clean`:: remove built files. If used like this +ninja -t clean+ it +removes all the built files. If used like this ++ninja -t clean _targets..._+ or like this ++ninja -t clean target _targets..._+ it removes the given targets and +recursively all files built for it. If used like this ++ninja -t clean rule _rules_+ it removes all files built using the given +rules. The depfiles are not removed. Files created but not referenced in +the graph are not removed. This tool takes in account the +-v+ and the ++-n+ options (note that +-n+ implies +-v+). It returns non-zero if an +error occurs. + +Ninja file reference +-------------------- + +A file is a series of declarations. A declaration can be one of: + +1. A rule declaration, which begins with +rule _rulename_+, and + then has a series of indented lines defining variables. + +2. A build edge, which looks like +build _output1_ _output2_: + _rulename_ _input1_ _input2_+. + + Implicit dependencies may be tacked on the end with +| + _dependency1_ _dependency2_+. + + Order-only dependencies may be tacked on the end with +|| + _dependency1_ _dependency2_+. (See <<ref_dependencies,the reference on + dependency types>>.) + + +3. Variable declarations, which look like +_variable_ = _value_+. + +4. References to more files, which look like +subninja _path_+ or + +include _path_+. The difference between these is explained below + <<ref_scope,in the discussion about scoping>>. + +Comments begin with `#` and extend to the end of the line. + +Newlines are significant, but they can be escaped by putting a `\` +before them. + +Other whitespace is only significant if it's at the beginning of a +line. If a line is intended more than the previous one, it's +considered part of its parent's scope; if it is indented less than the +previous one, it closes the previous scope. + +Rule variables +~~~~~~~~~~~~~~ +[[ref_rule]] + +A `rule` block contains a list of `key = value` declarations that +affect the processing of the rule. Here is a full list of special +keys. + +`command` (_required_):: the command line to run. This string (after + $variables are expanded) is passed directly to `sh -c` without + interpretation by Ninja. + +`depfile`:: path to an optional `Makefile` that contains extra + _implicit dependencies_ (see <<ref_dependencies,the reference on + dependency types>>). This is explicitly to support `gcc` and its `-M` + family of flags, which output the list of headers a given `.c` file + depends on. ++ +Use it like in the following example: ++ +---- +rule cc + depfile = $out.d + command = gcc -MMD -MF $out.d [other gcc flags here] +---- ++ +When loading a `depfile`, Ninja implicitly adds edges such that it is +not an error if the listed dependency is missing. This allows you to +delete a depfile-discovered header file and rebuild, without the build +aborting due to a missing input. + + +`description`:: a short description of the command, used to pretty-print + the command as it's running. The `-v` flag controls whether to print + the full command or its description; if a command fails, the full command + line will always be printed before the command's output. + +Additionally, the special `$in` and `$out` variables expand to the +space-separated list of files provided to the `build` line referencing +this `rule`. + +Build dependencies +~~~~~~~~~~~~~~~~~~ +[[ref_dependencies]] + +There are three types of build dependencies which are subtly different. + +1. _Explicit dependencies_, as listed in a build line. These are + available as the `$in` variable in the rule. Changes in these files + cause the output to be rebuilt; if these file are missing and + Ninja doesn't know how to build them, the build is aborted. ++ +This is the standard form of dependency to be used for e.g. the +source file of a compile command. + +2. _Implicit dependencies_, either as picked up from + a `depfile` attribute on a rule or from the syntax +| _dep1_ + _dep2_+ on the end of a build line. The semantics are identical to + explicit dependencies, the only difference is that implicit dependencies + don't show up in the `$in` variable. ++ +This is for expressing dependencies that don't show up on the +command line of the command; for example, for a rule that runs a +script, the script itself should be an implicit dependency, as +changes to the script should cause the output to rebuild. ++ +Note that dependencies as loaded through depfiles have slightly different +semantics, as described in the <<ref_rule,rule reference>>. + +3. _Order-only dependencies_, expressed with the syntax +|| _dep1_ + _dep2_+ on the end of a build line. When these are missing, the + output is not rebuilt until they are built, but once they are + available further changes to the files do not affect the output. ++ +Order-only dependencies can be useful for bootstrapping dependencies +that are only discovered during build time: for example, to generate a +header file before starting a subsequent compilation step. (Once the +header is used in compilation, a generated dependency file will then +express the implicit dependency.) + +Evaluation and scoping +~~~~~~~~~~~~~~~~~~~~~~ +[[ref_scope]] + +Top-level variable declarations are scoped to the file they occur in. + +The `subninja` keyword, used to include another `.ninja` file, +introduces a new scope. The included `subninja` file may use the +variables from the parent file, and shadow their values for the file's +scope, but it won't affect values of the variables in the parent. + +To include another `.ninja` file in the current scope, much like a C +`#include` statement, use `include` instead of `subninja`. + +Variable declarations indented in a `build` block are scoped to the +`build` block. This scope is inherited by the `rule`. The full +lookup order for a variable referenced in a rule is: + +1. Rule-level variables (i.e. `$in`, `$command`). + +2. Build-level variables from the `build` that references this rule. + +3. File-level variables from the file that the `build` line was in. + +4. Variables from the file that included that file using the + `subninja` keyword. + +Future work +----------- + +Some pieces I'd like to add: + +_inotify_. I had originally intended to make Ninja be memory-resident +and to use `inotify` to keep the build state hot at all times. But +upon writing the code I found it was fast enough to run from scratch +each time. Perhaps a slower computer would still benefit from +inotify; the data structures are set up such that using inotify +shouldn't be hard. + +_build estimation and analysis_. As part of build correctness, Ninja +keeps a log of the time spent on each build statement. This log +format could be adjusted to instead store timing information across +multiple runs. From that, the total time necessary to build could be +estimated, allowing Ninja to print status like "3 minutes until +complete" when building. Additionally, a tool could output which +commands are the slowest or which directories take the most time +to build. + +_many others_. See the `todo` file included in the distribution. |