Non-Reproducible / Weird Uncertainty Results

[FloLins]

Hello,

I wanted to check the uncertainty properties of my SWAG models and can not reproduce the values from the paper.
When I train a VGG16/CIFAR10 model with the parameters from the paper and use your uncertainty script like this:

python uncertainty.py --data_path= --model=VGG16 --dataset=CIFAR10 --method=SWAG --scale=0.5 --use_test --cov_mat --file=<path/swag-300.pt> --save_path=<save_path>

It will result in a good accuracy, but in a huge NLL:

30/30
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 79/79 [00:00<00:00, 88.94it/s]
Accuracy: 0.9366
NLL: 1990.9390262566017

Thanks in advance

[wjmaddox]

i'm pretty sure that's close to the correct number but off by a scaling factor equal to the size of the dataset,

swa_gaussian/experiments/uncertainty/uncertainty.py

Line 214 in 61a918f

print("NLL:", nll(predictions / (i + 1), targets))

scaling factor is 10k, which produces a nll of 0.199, which is what's in fig 2 of the paper