|
|
Now, the unit tests are ran twice -- once with POST_BUILD (i.e. default mode)
and again with PRE_TEST (i.e. new discovery mode).
Both modes of setting gtest discovery mode are also tested:
1. Using the global override (i.e. CMAKE_GTEST_DISCOVER_TESTS_DISCOVERY_MODE)
2. Explicitly passing DISCOVERY_MODE in calls to gtest_discover_tests (in GoogleTestDiscoveryTimeout.cmake)
The goal is to show that the new PRE_TEST discovery mode does not break existing behavior
(i.e. should not break POST_BUILD mode) and should also pass the same tests
in the same way.
A few non trivial implementation details worth noting:
1. Refactoring discovery_timeout_test into own project
Originally, I tried doing:
```
run_GoogleTest(POST_BUILD)
run_GoogleTest(PRE_TEST)
```
Without changing the internal structure of run_GoogleTest.
But since discovery_timeout_test is part of the same project as the other tests,
and CTest include files always get evaluated and that's where test discovery occurs,
this means every other test now notices the timeout problem when running in PRE_TEST mode.
As a result, keeping the existing test structure meant that each existing test
(and any new test) would need to have its own PRE_TEST / POST_BUILD variant for stderr and stdout
in order to handle the case where discovery_timeout_test timed out.
This exponential increase in test output files introduced unnecessary complexity
and made it more cumbersome to work on test cases.
Why should an unrelated test case care about discovery_timeout_test?
So, to fix that issue, the tests were broken apart into two main groups:
1. run_GoogleTest_discovery_timeout (the test dealing with discovery_timeout_test)
2. run_GoogleTest (everything else)
This isolates the PRE_TEST / POST_BUILD timeout variants to a single test case.
And the other test cases remain unchanged -- further driving home the point that
DISCOVERY_MODE shouldn't change existing behavior.
2. Different number of PRE_TEST / POST_BUILD file variants
On the PRE_TEST path, different build systems / compilers (i.e. MSBuild and ninja/gcc)
produces different build output when building discovery_timeout_test,
but we don't actually care what it is, just as long as it builds
successfully.
This the fundamental difference in behavior between POST_BUILD (which would have failed)
and PRE_TEST (which doesn't) and is the reason why we don't need
a GoogleTest-discovery-build-result.txt or GoogleTest-discovery-build-stdout.txt
3. Fix flaky discovery timeout test
The test expects to see:
> Output:
> timeout
> case.
But sometimes, the test would only produce:
> Output:
> timout
In certain environments, specifically when built with OpenWatcom 1.4,
and while the build server was under heavy load (i.e. running many tests in parallel),
std::endl behaves inconsistently and doesn't completely
flush std::cout when the program is terminated due to timeout.
This results in inconsistent test failures because the actual output
doesn't fully match what's expected.
At first we tried adding an additional:
std::cout << std::flush
That didn't work. But using C-style printf() and fflush() appears to do
the trick:
> This time I managed to get on the machine while it was still busy doing other nightly builds
> and could reproduce the problem reliably. With that I was finally able to find a fix.
> It turns out my earlier hypothesis that C++ stream flushing was not working on the old compiler was correct,
> but even .flush() is not enough.
> I changed it to use C-style printf() and fflush() and now the test passes on that build.
> -- Brad King <brad.king@kitware.com>
Co-authored-by: Ryan Thornton <ThorntonRyan@JohnDeere.com>
Co-authored-by: Kevin Puetz <PuetzKevinA@JohnDeere.com>
|
