Bazel Remote Cache: TLS tests, basic auth tests failing

[comius]

• TLS tests started failing on 2021-05-07
• basic auth tests started failing on 2021-05-05, because of a missing test file, since 2021-05-07 same as TLS tests
• autosheriff blames infra

Link: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/2025#ee9eaee2-0f65-4a58-abb2-cb05bba3b690

Example failure:

WARNING: Writing to Remote Cache:
Error while uploading artifact with digest '5e9f615a82e4dca1e09d9386b774d6c886de22b1395390e3b6988daf91d117bd/144'
Target //:bazel-remote up-to-date:
  bazel-out/k8-fastbuild-ST-3891b650e255/bin/bazel-remote_/bazel-remote
INFO: Elapsed time: 4.230s, Critical Path: 2.74s
INFO: 456 processes: 436 remote cache hit, 19 internal, 1 linux-sandbox.
INFO: Build completed successfully, 456 total actions
INFO: Build completed successfully, 456 total actions
 
grep -A 1 "WARNING: Writing to Remote Cache:" "$tmpdir/unauthenticated_write.log" | \
	tr '\n' '|' > "$tmpdir/unauthenticated_write.log.singleline"
if ! grep --silent "WARNING: Writing to Remote Cache:|BulkTransferException|" "$tmpdir/unauthenticated_write.log.singleline"
then
	# We seem to always have one cache miss with a rebuild.
	# So we expect a single cache write attempt, and it should fail.
	echo "Error: expected a warning when writing to the remote cache fails"
	exit 1
fi
Error: expected a warning when writing to the remote cache fails
Traceback (most recent call last):
  File "bazelci.py", line 3402, in <module>
    sys.exit(main())
  File "bazelci.py", line 3368, in main
    execute_commands(
  File "bazelci.py", line 1089, in execute_commands
    execute_shell_commands(task_config.get("shell_commands", None))
  File "bazelci.py", line 1485, in execute_shell_commands
    execute_command([shell_command], shell=True)
  File "bazelci.py", line 2079, in execute_command
    return subprocess.run(
  File "/usr/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['set -e\necho +++ Run basic auth tests\n.bazelci/basic-auth-tests.sh']' returned non-zero exit status 1.

[philwo]

That's pretty mysterious. I think their test might be failing due to the error message having changed (probably just no longer prints the weird "BulkTransferException" thing, but a more user-friendly "Error while uploading artifact with digest [...]" message?), but Culprit Finder should have correctly found the commit that changed the error message then.

@coeuvre @meteorcloudy Any idea?

[coeuvre]

Yes, the message for upload errors is changed by bazelbuild/bazel@913a985 which is included by 4.1.0rc1.

[meteorcloudy]

I think autosheriff is correct this time, it's caused by the change of presubmit.yml of Bazel Remote Cache the project:
buchgr/bazel-remote@f7f9c9d

Looks like this change works with Bazel 4.0 but not with 4.1 or HEAD due to bazelbuild/bazel@913a985

/cc @mostynb

[mostynb]

Thanks for the heads-up. I will take a look.

[mostynb]

I have disabled this fragile test in bazel-remote.

[meteorcloudy]

Thank you!

[comius]

Thanks!