Test KokkosKernels_sparse_openmp_MPI_1 still (randomly) timing out in in build Trilinos-atdm-white-ride-gnu-debug-openmp on 'white'

[bartlettroscoe]

@trilinos/kokkos-kernels, @srajama1 (Trilinos Linear Solver Product Lead)

Next Action Status

PR #3173 disabled this test in this build. There have been no timeouts of this test on any system since 9/21/2018 except in the the build Trilinos-atdm-mutrino-intel-opt-openmp-KNL (which is being handled #3864).

Description

As shown in this query the test KokkosKernels_sparse_openmp_MPI_1 timed out at 10 minutes in the build Trilinos-atdm-white-ride-gnu-debug-openmp on 'white'. But as one can see, the test was taking upwards of nearly 10 minutes to complete before in this build going back to 7/1/2018.

Steps to Reproduce

Use the build name gnu-debug-openmp (not cuda-debug) on the machine 'white' and enable the package KokkosKernels (not MueLu) using the commands shown at:

• https:/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#ridewhite

NOTE: One cannot currently reproduce this on 'ride' because of the upgrade of 'white' but not 'ride'. See TRIL-215.

[bartlettroscoe]

Looks like the efforts in #2827 to selectively disable some individual unit tests in KokkosKernels_sparse_openmp_MPI_1 have not been successful to drop the runtime for this test on this build on this platform low enough to avoid timeouts.

Looking at all of the platforms where this test was run yesterday at where this test is getting run in a debug-openmp build, this is the only build where the test is taking close to 10 minutes. It is running and passing pretty quickly in intel-debug-openmp and gnu-debug-openmp builds on several other platforms as shown in this query. Therefore, I would suggest it is not much of a loss to disable this test in this build on 'white'. Besides, 'white' and 'ride' are supposed to be ATS-2 semi-clones of Sierra which uses GPUs and CUDA as the workhorse, so why do we care about OpenMP builds on that system anyway?

[mhoemmen]

so why do we care about OpenMP builds on that system anyway?

It's pretty likely that many apps won't find it worthwhile to run on the GPUs, so it would be nice to test OpenMP a little bit :-)

[bartlettroscoe]

PR #3173 was merged which disables this test. I will now put this into review and what till tomorrow 7/24/2018 to make sure it is disabled.

[fryeguy52]

this query shows that the last time the test was run for this build was on 7/22/2018 which means it was disabled. closing this issue

[bartlettroscoe]

@fryeguy52, this same test is timing out in the same build gnu-debug-openmp on 'waterman' as shown here. Therefore, I think we need to disable this test for that build as well.

I will reopen this issue just as a reminder to do that.

[bartlettroscoe]

@fryeguy52, actually, our policy is not close GitHub issues if we disable tests but to just put the "Disabled Tests" label on them and leave them open.

[bartlettroscoe]

As shown in this query, this test has only failed or timed-out in the 'mutrino' builds Trilinos-atdm-mutrino-intel-opt-openmp-HSW and Trilinos-atdm-mutrino-intel-opt-openmp-KNL since 9/21/2018.

The timeouts in the build Trilinos-atdm-mutrino-intel-opt-openmp-KNL are being addressed in #3864. The one failures in the build Trilinos-atdm-mutrino-intel-opt-openmp-HSW shown here was a problem with SLURM on 'mutrino' showing the error:

srun: fatal: Invalid user id: 84966

The failure (not timeout) in the build shown here showed that same 'mutrino' SLURM error:

srun: fatal: Invalid user id: 84966

Therefore, since there have been no timeouts of this test on any system since 9/21/2018 except in the the build Trilinos-atdm-mutrino-intel-opt-openmp-KNL (which is being handled #3864), I think it is safe to close this issue.