metamorphic/crossversion_test: failed

[cockroach-teamcity]

. failed with artifacts on refs/heads/master @ 41145ee9d232:

[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 281s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 341s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 401s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 461s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 521s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 581s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 641s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 701s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 761s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 821s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 881s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 941s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 1001s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 1061s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 1121s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 1181s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 1241s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 1301s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 1361s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 1421s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 1481s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 1541s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 1601s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 1661s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 1721s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 1781s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 1841s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 1901s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 1961s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 2021s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 2081s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 2141s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 2201s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 2261s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 2321s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 2381s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 2441s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 2501s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 2561s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 2621s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 2681s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 2742s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 2802s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 2862s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 2922s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 2982s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 3042s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 3102s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/crossversion:crossversion_test; 3162s processwrapper-sandbox
[677 / 678] Testing @@com_github_cockroachdb_pebble//internal/metamorphic/c

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: PEBBLE-267

[aadityasondhi]

We timed out during the download operation:

 panic: operation took longer than 4m0s: db1.Download("exdrpxlgo@197" /* start */, "qrryqcegnhxy@146" /* end */, false /* viaBackingFileDownload */, "egdr@101" /* start */, "xklc@12" /* end */, true /* viaBackingFileDownload */, "jahykdetegy@2" /* start */, "wuaibbvqybv@173" /* end */, true /* viaBackingFileDownload */)

Likely a flake. Will bump the timeout for that operation to avoid this flake.

[jbowens]

Do we understand why this took 4m? My understanding is that in the test there is no dependency on the network and the "download" operation is entirely local. I don't think it's a release blocker regardless since it's limited to online restore, but I worry there might be a bug lingering somewhere that causes deadlock.

[aadityasondhi]

Do we understand why this took 4m?

No, I can try digging into the logs.