Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make some tests use deterministic torch algorithms #108

Merged
merged 3 commits into from
Jul 25, 2023

Conversation

RaulPPelaez
Copy link
Contributor

This fixes some tests not passing on some architectures due to numerical inaccuraccies (at least on my machine).

@RaulPPelaez RaulPPelaez mentioned this pull request Jul 25, 2023
@RaulPPelaez
Copy link
Contributor Author

@mikemhenry @raimis , try the GPU runner here please.

@mikemhenry
Copy link
Collaborator

Tests running here https:/openmm/NNPOps/actions/runs/5660607367

@mikemhenry
Copy link
Collaborator

RuntimeError: Deterministic behavior was enabled with either `torch.use_deterministic_algorithms(True)` or `at::Context::setDeterministicAlgorithms(true)`, but this operation is not deterministic because it uses CuBLAS and you have CUDA >= 10.2. To enable deterministic behavior in this case, you must set an environment variable before running your PyTorch application: CUBLAS_WORKSPACE_CONFIG=:4096:8 or CUBLAS_WORKSPACE_CONFIG=:16:8. For more information, go to https://docs.nvidia.com/cuda/cublas/index.html#cublasApi_reproducibility

Looks like I need to add this to the runner

@mikemhenry
Copy link
Collaborator

New test here https:/openmm/NNPOps/actions/runs/5661251365

@mikemhenry
Copy link
Collaborator

From here https://docs.nvidia.com/cuda/cublas/index.html#results-reproducibility I chose the export CUBLAS_WORKSPACE_CONFIG=:4096:8 since I don't think we are tight on GPU memory

@mikemhenry
Copy link
Collaborator

Tests pass!

@mikemhenry mikemhenry enabled auto-merge (squash) July 25, 2023 21:03
@mikemhenry
Copy link
Collaborator

I setup auto-merge, review required, and status checks. Let me know if that causes any issues @RaulPPelaez @raimis

@mikemhenry mikemhenry merged commit d15cb91 into openmm:master Jul 25, 2023
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants