TeuchosCore_testTeuchosTestForTermination_X_MPI_4 tests randomly timing out/hanging in new cee-rhel6 openmpi-4.0.1 builds starting 2020-01-04

[bartlettroscoe]

CC: @trilinos/teuchos, @bartlettroscoe

Next Action Status

Likely due to a defect in this OpenMPI 4.0.1 install not correctly terminating MPI job when one of the ranks is terminated. Next: Let run for a few weeks to get some more data then decide what to do.

Description

As shown in this query the tests:

• TeuchosCore_testTeuchosTestForTermination_0_MPI_4
• TeuchosCore_testTeuchosTestForTermination_2_MPI_4
• TeuchosCore_testTeuchosTestForTermination_3_MPI_4

in the new builds:

• Trilinos-atdm-cee-rhel6_clang-5.0.1_openmpi-4.0.1_serial_static_opt
• Trilinos-atdm-cee-rhel6_gnu-7.2.0_openmpi-4.0.1_serial_shared_opt

are randomly timing out.

These are most likely hanging when one of the processes calls abort but it seems that OpenMPI 4.0.1 is not properly terminating the other processes.

Current Status on CDash

• Failing TeuchosCore_testTeuchosTestForTermination_X_MPI_4 tests in these builds over last 2 weeks

Steps to Reproduce

One should be able to reproduce this failure on any CEE RHEL6 or RHEL7 machine as described in:

• https:/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md

More specifically, the commands given for the system 'cee-rhel6' are provided at:

• https:/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#cee-rhel6-environment

The exact commands to reproduce this issue should be:

$ cd <some_build_dir>/

$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh <build-name>

$ cmake \
 -GNinja \
 -DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
 -DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Teuchos=ON \
 $TRILINOS_DIR

$ make NP=16

$ ctest -j12

[bartlettroscoe]

I added the label ATDM Env Issue because I believe this is a defect in this OpenMPI 4.0.1 installation in not aborting other ranks when one of the ranks aborts.

I will let this test keep running in these builds so that we can get some more statistics.

[bartlettroscoe]

Also, I consider this to be a nonblocking issue since as long as your program does not terminate on a process this all is good. And if it does terminate on a process, the MPI job will just hang (until the timeout is reached).

[bartlettroscoe]

As shown in this query these tests started passing when the 'cee-rhel6' builds were switched from using 'openmpi-4.0.1' to 'openmpi-4.0.2' on testing day 2020-01-16. Looks like that version of OpenMPI fixed the behavior of MPI_Abort() :-)

Therefore, this is fixed. Closing as fixed :-)