Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spurious failure in incremental\cache_file_headers.rs on Windows #38620

Closed
alexcrichton opened this issue Dec 26, 2016 · 9 comments
Closed

Spurious failure in incremental\cache_file_headers.rs on Windows #38620

alexcrichton opened this issue Dec 26, 2016 · 9 comments
Labels
A-spurious Area: Spurious failures in builds (spuriously == for no apparent reason) O-windows Operating system: Windows

Comments

@alexcrichton
Copy link
Member

alexcrichton commented Dec 26, 2016

I've seen this failure a number of times on AppVeyor at this point:

Check compiletest suite=incremental mode=incremental (x86_64-pc-windows-msvc -> x86_64-pc-windows-msvc)
running 69 tests
F....................................................................
failures:
---- [incremental] incremental\cache_file_headers.rs stdout ----
	
error in revision `rpass2`: compilation failed!
status: exit code: 101
command: PATH="C:\projects\rust\build\x86_64-pc-windows-msvc\stage2\bin;C:\Program Files (x86)\Windows Kits\10\bin\x64;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\amd64;C:\projects\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\release\deps;C:\projects\rust\build\x86_64-pc-windows-msvc\stage2-rustc\x86_64-pc-windows-msvc\release\deps;C:\projects\rust\build\x86_64-pc-windows-msvc\stage2-test\x86_64-pc-windows-msvc\release\deps;C:\projects\rust\build\x86_64-pc-windows-msvc\stage2-std\x86_64-pc-windows-msvc\release\deps;C:\msys64\mingw64\bin;C:\msys64\mingw64\bin;C:\msys64\usr\bin;C:\Perl\site\bin;C:\Perl\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Program Files\7-Zip;C:\Program Files\Microsoft\Web Platform Installer;C:\Tools\GitVersion;C:\Tools\PsTools;C:\Program Files\Git LFS;C:\Program Files\Mercurial;C:\Program Files (x86)\Subversion\bin;C:\Program Files\Microsoft SQL Server\120\Tools\Binn;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\110\Tools\Binn;C:\Program Files (x86)\Microsoft SQL Server\120\Tools\Binn;C:\Program Files\Microsoft SQL Server\120\DTS\Binn;C:\Program Files (x86)\Microsoft SQL Server\120\Tools\Binn\ManagementStudio;C:\Program Files\Microsoft Windows Performance Toolkit;C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit;C:\Tools\WebDriver;C:\Program Files (x86)\Microsoft SDKs\TypeScript\1.4;C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\PrivateAssemblies;C:\Program Files (x86)\Microsoft SDKs\Azure\CLI\wbin;C:\Ruby193\bin;C:\Tools\NUnit\bin;C:\Tools\xUnit;C:\Tools\MSpec;C:\Tools\Coverity\bin;C:\Program Files (x86)\CMake\bin;C:\go\bin;C:\Program Files\Java\jdk1.8.0\bin;C:\Python27;C:\Program Files\erl7.3\bin;C:\Program Files\nodejs;C:\Program Files (x86)\iojs;C:\Program Files\iojs;C:\Users\appveyor\AppData\Roaming\npm;C:\Program Files\Microsoft SQL Server\130\Tools\Binn;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit;C:\Program Files (x86)\MSBuild\14.0\Bin;C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\Extensions\Microsoft\SQLDB\DAC\120;C:\Tools\NuGet;C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow;C:\Program Files\Amazon\AWSCLI;C:\Windows\SysWOW64\WindowsPowerShell\v1.0\Modules\TShell\TShell;C:\Program Files\Microsoft DNX\Dnvm;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\130\Tools\Binn;C:\Program Files\Git\cmd;C:\Program Files\Git\usr\bin;C:\Program Files (x86)\Microsoft SQL Server\130\Tools\Binn;C:\Program Files (x86)\Microsoft SQL Server\130\DTS\Binn;C:\Program Files\Microsoft SQL Server\130\DTS\Binn;C:\Program Files (x86)\Microsoft SQL Server\110\DTS\Binn;C:\Program Files (x86)\Microsoft SQL Server\120\DTS\Binn;C:\Program Files (x86)\Apache\Maven\bin;C:\Program Files\LLVM\bin;C:\ProgramData\chocolatey\bin;C:\Program Files\Microsoft Service Fabric\bin\Fabric\Fabric.Code;C:\Program Files\Microsoft SDKs\Service Fabric\Tools\ServiceFabricLocalClusterManager;C:\Python27\Scripts;C:\Program Files (x86)\Yarn\bin;C:\Tools\NUnit3;C:\Program Files\dotnet;C:\Program Files (x86)\nodejs;C:\Users\appveyor\.dnx\runtimes\dnx-clr-win-x86.1.0.0-rc1-update2\bin;C:\Users\appveyor\AppData\Local\Yarn\.bin;C:\Users\appveyor\AppData\Roaming\npm;C:\Program Files\AppVeyor\BuildAgent;C:\projects\rust\sccache2" C:\projects\rust\build\x86_64-pc-windows-msvc\stage2\bin\rustc.exe C:\projects\rust\src/test\incremental\cache_file_headers.rs -L C:\projects\rust\build\x86_64-pc-windows-msvc\test\incremental --target=x86_64-pc-windows-msvc --cfg rpass2 -Z incremental=C:\projects\rust\build\x86_64-pc-windows-msvc\test\incremental\cache_file_headers.inc --error-format json -L C:\projects\rust\build\x86_64-pc-windows-msvc\test\incremental\cache_file_headers.stage2-x86_64-pc-windows-msvc.incremental.libaux -C prefer-dynamic -o C:\projects\rust\build\x86_64-pc-windows-msvc\test\incremental\cache_file_headers.stage2-x86_64-pc-windows-msvc.exe -Crpath -O -Lnative=C:\projects\rust\build\x86_64-pc-windows-msvc\rust-test-helpers -Zincremental-info
stdout:
------------------------------------------
incremental: session directory: 4 files hard-linked
incremental: session directory: 0 files copied
incremental: ignoring cache artifact `dep-graph.bin`: Different compiler version
incremental: re-using 0 out of 1 modules
------------------------------------------
stderr:
------------------------------------------
{"message":"linking with `link.exe` failed: exit code: 1104","code":null,"level":"error","spans":[],"children":[{"message":"\"C:\\\\Program Files (x86)\\\\Microsoft Visual Studio 14.0\\\\VC\\\\bin\\\\amd64\\\\link.exe\" \"/LIBPATH:C:\\\\Program Files (x86)\\\\Microsoft Visual Studio 14.0\\\\VC\\\\lib\\\\amd64\" \"/LIBPATH:C:\\\\Program Files (x86)\\\\Windows Kits\\\\10\\\\lib\\\\10.0.14393.0\\\\ucrt\\\\x64\" \"/LIBPATH:C:\\\\Program Files (x86)\\\\Windows Kits\\\\10\\\\lib\\\\10.0.14393.0\\\\um\\\\x64\" \"/NOLOGO\" \"/NXCOMPAT\" \"/LIBPATH:C:\\\\projects\\\\rust\\\\build\\\\x86_64-pc-windows-msvc\\\\stage2\\\\lib\\\\rustlib\\\\x86_64-pc-windows-msvc\\\\lib\" \"C:\\\\projects\\\\rust\\\\build\\\\x86_64-pc-windows-msvc\\\\test\\\\incremental\\\\cache_file_headers.cache_file_headers.o\" \"/OUT:C:\\\\projects\\\\rust\\\\build\\\\x86_64-pc-windows-msvc\\\\test\\\\incremental\\\\cache_file_headers.stage2-x86_64-pc-windows-msvc.exe\" \"/OPT:REF,ICF\" \"/DEBUG\" \"/LIBPATH:C:\\\\projects\\\\rust\\\\build\\\\x86_64-pc-windows-msvc\\\\test\\\\incremental\" \"/LIBPATH:C:\\\\projects\\\\rust\\\\build\\\\x86_64-pc-windows-msvc\\\\test\\\\incremental\\\\cache_file_headers.stage2-x86_64-pc-windows-msvc.incremental.libaux\" \"/LIBPATH:C:\\\\projects\\\\rust\\\\build\\\\x86_64-pc-windows-msvc\\\\rust-test-helpers\" \"/LIBPATH:C:\\\\projects\\\\rust\\\\build\\\\x86_64-pc-windows-msvc\\\\stage2\\\\lib\\\\rustlib\\\\x86_64-pc-windows-msvc\\\\lib\" \"/LIBPATH:C:\\\\projects\\\\rust\\\\build\\\\x86_64-pc-windows-msvc\\\\stage2\\\\lib\\\\rustlib\\\\x86_64-pc-windows-msvc\\\\lib\" \"std-5a02bf719865b808.dll.lib\" \"C:\\\\projects\\\\rust\\\\build\\\\x86_64-pc-windows-msvc\\\\stage2\\\\lib\\\\rustlib\\\\x86_64-pc-windows-msvc\\\\lib\\\\libcompiler_builtins-b05c732613224206.rlib\" \"advapi32.lib\" \"ws2_32.lib\" \"userenv.lib\" \"shell32.lib\" \"msvcrt.lib\"","code":null,"level":"note","spans":[],"children":[],"rendered":null},{"message":"LINK : fatal error LNK1104: cannot open file 'C:\\projects\\rust\\build\\x86_64-pc-windows-msvc\\test\\incremental\\cache_file_headers.stage2-x86_64-pc-windows-msvc.exe'\r\n","code":null,"level":"note","spans":[],"children":[],"rendered":null}],"rendered":null}
{"message":"aborting due to previous error","code":null,"level":"error","spans":[],"children":[],"rendered":null}
------------------------------------------
thread '[incremental] incremental\cache_file_headers.rs' panicked at 'explicit panic', src\tools\compiletest\src\runtest.rs:2427
note: Run with `RUST_BACKTRACE=1` for a backtrace.
failures:
    [incremental] incremental\cache_file_headers.rs
test result: FAILED. 68 passed; 1 failed; 0 ignored; 0 measured

where the actual error is:

LINK : fatal error LNK1104: cannot open file 'C:\\projects\\rust\\build\\x86_64-pc-windows-msvc\\test\\incremental\\cache_file_headers.stage2-x86_64-pc-windows-msvc.exe

So far I've been unable to isolate what's happening here much less reproduce it, unfortunately

Other examples are:

Conclusions so far is that:

  • This is not MSVC or GNU specific
  • This is not 32 or 64-bit specific
  • It's always the cache_file_headers.rs test
  • This test is always the first failure in the incremental test suite
@alexcrichton alexcrichton added the A-spurious Area: Spurious failures in builds (spuriously == for no apparent reason) label Dec 26, 2016
@alexcrichton
Copy link
Member Author

cc @michaelwoerister

@michaelwoerister
Copy link
Member

This test relies on an environment variable being set correctly by compiletest. Might that be a hint?

@alexcrichton
Copy link
Member Author

Hm I think that may be a red herring. Presumably the env var is being set correctly because the compiler is recompiling, right?

Digging more into this this morning I came across https://bugzilla.mozilla.org/show_bug.cgi?id=509960, a similar error, but no resolution there. One possibility is disk space running out but this is such a deterministic error that seems unlikely.

@alexcrichton alexcrichton added the O-windows Operating system: Windows label Dec 27, 2016
@alexcrichton
Copy link
Member Author

My guess is that this is failing because something else has the file open. On Windows at least if there's an open file handle you're unable to delete that file. This is happening on both MSVC and GNU so I don't think it's linker specific (e.g. no problems like with mspdbsrv.exe I believe).

@michaelwoerister you wouldn't happen to have any ideas about when we'd open this file, would you? IIRC the compiler never actually opens the output file, it just relies on the linker to do so. I also couldn't find anything in the compiletest incremental pieces as well...

@brson
Copy link
Contributor

brson commented Dec 29, 2016

Why would this only show up on AppVeyor? Is the Windows job object set up differently in rustbuild than in rust-buildbot?

@alexcrichton
Copy link
Member Author

That's a good question! (I'm not sure why it's only on AppVeyor). We aren't actually using job objects on AppVeyor like we were on rust-buildbot, notably we aren't using rustjob. The rustbuild job object (which we do use), however, should be the same across rust-buildbot/AppVeyor.

Between AppVeyor and rust-buildbot we've had a ton of changes, though. AppVeyor has likely a newer install of Windows and also a newer version of Visual Studio.

@brson
Copy link
Contributor

brson commented Dec 29, 2016

It doesn't look like compiletest is doing anything itself that would hold the exe open.

alexcrichton added a commit to alexcrichton/rust that referenced this issue Dec 30, 2016
This commit is an attempt to debug rust-lang#38620 since we're unable to reproduce it
locally. It follows the [advice] of those with AppVeyor to use the `handle.exe`
tool to try to debug what processes have a handle to the file open.

This won't be guaranteed to actually help us, but hopefully it'll diagnose
something at some point?

[advice]: http://help.appveyor.com/discussions/questions/2898
bors added a commit that referenced this issue Dec 30, 2016
appveyor: Attempt to debug flaky test runs

This commit is an attempt to debug #38620 since we're unable to reproduce it
locally. It follows the [advice] of those with AppVeyor to use the `handle.exe`
tool to try to debug what processes have a handle to the file open.

This won't be guaranteed to actually help us, but hopefully it'll diagnose
something at some point?

[advice]: http://help.appveyor.com/discussions/questions/2898
alexcrichton added a commit to alexcrichton/rust that referenced this issue Dec 30, 2016
appveyor: Attempt to debug flaky test runs

This commit is an attempt to debug rust-lang#38620 since we're unable to reproduce it
locally. It follows the [advice] of those with AppVeyor to use the `handle.exe`
tool to try to debug what processes have a handle to the file open.

This won't be guaranteed to actually help us, but hopefully it'll diagnose
something at some point?

[advice]: http://help.appveyor.com/discussions/questions/2898
@michaelwoerister
Copy link
Member

I can't think of anything special that this test case. It will try to overwrite the executable for each revision, but all the other incremental test cases do that too.

@alexcrichton
Copy link
Member Author

This hasn't showed up in ~2 months, so presumed fix by... something!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-spurious Area: Spurious failures in builds (spuriously == for no apparent reason) O-windows Operating system: Windows
Projects
None yet
Development

No branches or pull requests

3 participants