diff options
author | Albert Cheng <acheng@hdfgroup.org> | 2010-08-11 18:12:45 (GMT) |
---|---|---|
committer | Albert Cheng <acheng@hdfgroup.org> | 2010-08-11 18:12:45 (GMT) |
commit | 063e4b2e2ebf9c7995f29c3f5e5bd6e7ac0e9ec7 (patch) | |
tree | 6f8720d1eca9297fe439fd0c2d36b78f67a5e66d /testpar | |
parent | 345a9a3c237cc987a9fc0772c804f1b816a954a8 (diff) | |
download | hdf5-063e4b2e2ebf9c7995f29c3f5e5bd6e7ac0e9ec7.zip hdf5-063e4b2e2ebf9c7995f29c3f5e5bd6e7ac0e9ec7.tar.gz hdf5-063e4b2e2ebf9c7995f29c3f5e5bd6e7ac0e9ec7.tar.bz2 |
[svn-r19230] Reset alarm_seconds back to 20 minutes.
Description:
honest3 v1.8 failed in parallel test. It got stuck in the same
testpar/testphdf5 subtest (cbhsssdrpio). This is an old problem.
Upon closer inspection, the testphdf5, when terminated, had clocked
up 1hr 9min 46 sec wall clock time. Honest1 system also sent a message
that an mpi process has used up 30+ CPU minutes which exceeded their login
node cpu time limit and they killed the process. I also did a hand-run
of testphdf5. All subtests before cbhsssdrpio completed in a few minutes.
Therefore, it is safe to say the majority of the 70 minutes of wall clock
time are spent in the sub-test cbhsssdrpio. It also used up lots of CPU
time. cbhsssdrpio is likely infinite looping.
Since MPI application is prone to infinite looping due to message deadlock,
the testphdf5 has a built-in protection to give each subtest at most 20 minutes
of wall-clock time to run. When the 20 minutes wall-clock time is exceeded,
the testphdf5 will attempt to terminate itself. This prevents unnecessary
CPU time consumption in infinite looping.
But that clock limit was changed to 30 and then 60 minutes. I should have
but failed to, noticed the change mentioned by Quincey. IMO, 20 wall clock
time is more than sufficient for each subtest of testphdf5 to complete.
If a subtest takes longer than 20 minutes, it is likely infinite looping.
Giving it more time will not help.
If a subtest of testphdf5 takes more than 20 minutes, it should be broken
down to small tests that will finish way under 20 minutes so that it is
much easier to see progress and identify any deadlock problems.
In view of this, I am changing the testphdf5 time limit back to 20 minutes.
This will at least stop the CPU TIME exceeding limits and annoying the
system administrators.
Maybe there could be a provision, such as environment variable like
$HDF5_ALARM_SECOND to modify the alarm duration on individual execution.
Even so, that should be used temporary to see if an execution just needs
a little more time.
Tested: just eyeballed as the change is trivia.
Diffstat (limited to 'testpar')
0 files changed, 0 insertions, 0 deletions