Re-enable compatibility with readonly CARGO_HOME

[alexcrichton]

Previously Cargo would attempt to work as much as possible with a
previously filled out CARGO_HOME, even if it was mounted as read-only.
In #6880 this was regressed as a few global locks and files were always
attempted to be opened in writable mode.

This commit fixes these issues by correcting two locations:

• First the global package cache lock has error handling to allow
  acquiring the lock in read-only mode inaddition to read/write mode. If
  the read/write mode failed due to an error that looks like a readonly
  filesystem then we assume everything in the package cache is readonly
  and we switch to just acquiring any lock, this time a shared readonly
  one. We in theory aren't actually doing any synchronization at that
  point since it's all readonly anyway.

• Next when unpacking package we're careful to issue a stat call
  before opening a file in writable mode. This way our preexisting guard
  to return early if a package is unpacked will succeed before we open
  anything in writable mode.

Closes #6928

[rust-highfive]

r? @nrc

(rust_highfive has picked a reviewer for you, use r? to override)

[alexcrichton]

For posterity, the way I tested this was:

$ cargo build --features vendored-openssl
$ docker run \
  -v `rustc --print sysroot`:/rust:ro \
  -e RUSTC=/rust/bin/rustc \
  -w /src \
  -it \
  -v `pwd`:/src \
  -v $HOME/.cargo:/cargo:ro \
  -e CARGO_HOME=/cargo \
  ubuntu:18.04 \
  ./target/debug/cargo build

If that progresses to the point that it fails the build due to a missing cc then it means we got past the resolution phase and we've actually started executing rustc and everything is downloaded. I hit the two errors fixed here previously and afterwards it made more progress.

I'm not entirely certain if this matches up with Crater's use case, so I'm not 100% certain it will solve it.

[alexcrichton]

Oh right there's these things called permission bits, let me add a test.

[alexcrichton]

r? @ehuss

[ehuss]

@bors r+

[bors]

📌 Commit 5d9383e has been approved by ehuss

[bors]

⌛ Testing commit 5d9383e with merge 414c1eb...

[bors]

☀️ Test successful - checks-travis, status-appveyor
Approved by: ehuss
Pushing 414c1eb to master...

[djahandarie]

Is this PR a full solution to the issue given NixOS/nixpkgs#61618?

[jmrgibson]

Does this mean the cache should work in read-only mode too? We've got rust builds running in CI/CD, we'd like the rust toolchain directory and cargo binaries (eg cargo, fmt, etc) to be read-only for the service account running the builds, but at the moment that gets messed up because the cache directory is the same as CARGO_HOME.