Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Official builds are broken because of OS (RID, ...) property updates #48647

Closed
ViktorHofer opened this issue Feb 23, 2021 · 9 comments
Closed

Official builds are broken because of OS (RID, ...) property updates #48647

ViktorHofer opened this issue Feb 23, 2021 · 9 comments
Labels
area-Infrastructure
Milestone
6.0.0

Comments

@ViktorHofer
Copy link
Member 

D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'runtime.linux-arm.Microsoft.NETCore.ILAsm' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'runtime.linux-arm.Microsoft.NETCore.ILDAsm' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'runtime.linux-arm.Microsoft.NETCore.TestHost' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'runtime.linux-arm.runtime.native.System.IO.Ports' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'runtime.linux-arm64.Microsoft.NETCore.ILAsm' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'runtime.linux-arm64.Microsoft.NETCore.ILDAsm' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'runtime.linux-arm64.Microsoft.NETCore.TestHost' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'runtime.linux-arm64.runtime.native.System.IO.Ports' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'assets/symbols/runtime.linux-arm.Microsoft.NETCore.ILAsm.6.0.0-preview.3.21123.3.symbols.nupkg' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'assets/symbols/runtime.linux-arm.Microsoft.NETCore.ILDAsm.6.0.0-preview.3.21123.3.symbols.nupkg' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'assets/symbols/runtime.linux-arm.Microsoft.NETCore.TestHost.6.0.0-preview.3.21123.3.symbols.nupkg' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'assets/symbols/runtime.linux-arm.runtime.native.System.IO.Ports.6.0.0-preview.3.21123.3.symbols.nupkg' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'assets/symbols/runtime.linux-arm64.Microsoft.NETCore.ILAsm.6.0.0-preview.3.21123.3.symbols.nupkg' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'assets/symbols/runtime.linux-arm64.Microsoft.NETCore.ILDAsm.6.0.0-preview.3.21123.3.symbols.nupkg' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'assets/symbols/runtime.linux-arm64.Microsoft.NETCore.TestHost.6.0.0-preview.3.21123.3.symbols.nupkg' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'assets/symbols/runtime.linux-arm64.runtime.native.System.IO.Ports.6.0.0-preview.3.21123.3.symbols.nupkg' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : InvalidOperationException: Duplicate entries are not allowed for publishing to BAR, as this can cause race conditions and unexpected behavior
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error :    at Microsoft.DotNet.Maestro.Tasks.PushMetadataToBuildAssetRegistry.MergeBuildManifests(List`1 buildsMetadata) in /_/src/Maestro/Microsoft.DotNet.Maestro.Tasks/src/PushMetadataToBuildAssetRegistry.cs:line 519
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error :    at Microsoft.DotNet.Maestro.Tasks.PushMetadataToBuildAssetRegistry.PushMetadataAsync(CancellationToken cancellationToken) in /_/src/Maestro/Microsoft.DotNet.Maestro.Tasks/src/PushMetadataToBuildAssetRegistry.cs:line 96
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : 
    0 Warning(s)
    17 Error(s)

Caused by #48505 (it's the only change in the failing official build).
Official build (internal only): https://dev.azure.com/dnceng/internal/_build/results?buildId=1007559&view=logs&j=5c24930e-0ae1-546c-7353-8c3b261a7651&t=15a0e93a-4b94-56d6-ccae-6c186febb6f4&l=42

@am11 can you please take a look?
@ViktorHofer ViktorHofer added this to the 6.0.0 milestone Feb 23, 2021
@ghost
Copy link

ghost commented Feb 23, 2021

Tagging subscribers to this area: @ViktorHofer
See info in area-owners.md if you want to be subscribed.

Issue Details 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'runtime.linux-arm.Microsoft.NETCore.ILAsm' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'runtime.linux-arm.Microsoft.NETCore.ILDAsm' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'runtime.linux-arm.Microsoft.NETCore.TestHost' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'runtime.linux-arm.runtime.native.System.IO.Ports' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'runtime.linux-arm64.Microsoft.NETCore.ILAsm' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'runtime.linux-arm64.Microsoft.NETCore.ILDAsm' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'runtime.linux-arm64.Microsoft.NETCore.TestHost' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'runtime.linux-arm64.runtime.native.System.IO.Ports' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'assets/symbols/runtime.linux-arm.Microsoft.NETCore.ILAsm.6.0.0-preview.3.21123.3.symbols.nupkg' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'assets/symbols/runtime.linux-arm.Microsoft.NETCore.ILDAsm.6.0.0-preview.3.21123.3.symbols.nupkg' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'assets/symbols/runtime.linux-arm.Microsoft.NETCore.TestHost.6.0.0-preview.3.21123.3.symbols.nupkg' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'assets/symbols/runtime.linux-arm.runtime.native.System.IO.Ports.6.0.0-preview.3.21123.3.symbols.nupkg' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'assets/symbols/runtime.linux-arm64.Microsoft.NETCore.ILAsm.6.0.0-preview.3.21123.3.symbols.nupkg' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'assets/symbols/runtime.linux-arm64.Microsoft.NETCore.ILDAsm.6.0.0-preview.3.21123.3.symbols.nupkg' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'assets/symbols/runtime.linux-arm64.Microsoft.NETCore.TestHost.6.0.0-preview.3.21123.3.symbols.nupkg' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : Repeated Asset entry: 'assets/symbols/runtime.linux-arm64.runtime.native.System.IO.Ports.6.0.0-preview.3.21123.3.symbols.nupkg' - '6.0.0-preview.3.21123.3' 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : InvalidOperationException: Duplicate entries are not allowed for publishing to BAR, as this can cause race conditions and unexpected behavior
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error :    at Microsoft.DotNet.Maestro.Tasks.PushMetadataToBuildAssetRegistry.MergeBuildManifests(List`1 buildsMetadata) in /_/src/Maestro/Microsoft.DotNet.Maestro.Tasks/src/PushMetadataToBuildAssetRegistry.cs:line 519
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error :    at Microsoft.DotNet.Maestro.Tasks.PushMetadataToBuildAssetRegistry.PushMetadataAsync(CancellationToken cancellationToken) in /_/src/Maestro/Microsoft.DotNet.Maestro.Tasks/src/PushMetadataToBuildAssetRegistry.cs:line 96
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21105.12\tools\SdkTasks\PublishBuildAssets.proj(43,5): error : 
    0 Warning(s)
    17 Error(s)

Caused by #48505 (it's the only change in the failing official build).
Official build (internal only): https://dev.azure.com/dnceng/internal/_build/results?buildId=1007559&view=logs&j=5c24930e-0ae1-546c-7353-8c3b261a7651&t=15a0e93a-4b94-56d6-ccae-6c186febb6f4&l=42

@am11 can you please take a look?

Author: ViktorHofer
Assignees: -
Labels:

area-Infrastructure, blocking-official-build
Milestone: 6.0.0

@dotnet-issue-labeler dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Feb 23, 2021
@ViktorHofer ViktorHofer removed the untriaged New issue has not been triaged by the area owner label Feb 23, 2021
@runfoapp runfoapp bot mentioned this issue Feb 23, 2021
Closed
@ViktorHofer
Copy link
Member Author

ViktorHofer commented Feb 23, 2021 •
edited
Loading

This happens when multiple legs produce identical assets (identical based on their PackageId and PackageVersion).

The affected packages are:

  • Microsoft.NETCore.ILAsm
  • Microsoft.NETCore.ILDAsm
  • Microsoft.NETCore.TestHost
  • runtime.native.System.IO.Ports

And the colliding configurations are:

  • arm <--> musl arm
  • arm64 <--> musl arm64

The Linux arm leg produces the following assets:
image

The Linux musl arm leg produces the same assets:
image

Does that help meanwhile?

@ViktorHofer, can we have a officialbuild-like leg in public CI to make it easier to investigate? Repeated Asset entry sounds like some kind of a validation warning reported as error. I don't know how/what it is checking.

@dotnet/dnceng for ideas how to validate this scenario in public CI.

@mmitche
Copy link
Member

mmitche commented Feb 23, 2021

Repeated asset entry is the correct check in the official build. It ensures that the same asset wasn't published twice (for this exact reason), and it's an error. The most common case for this are where two different legs build the same non OS specific asset (e.g. System.IO.Pipelines), ending up with a non-deterministic build

Really tough to validate in public CI, since (by design) you have no join point where you would know about artifacts from all the inputs. Maybe you could make a set of validation rules for package names?

@mmitche
Copy link
Member

mmitche commented Feb 23, 2021

Honestly, I think that based on the number of times this hits (it's very infrequent), not validating it in public PRs is fine. We don't validate the entire test set in a public PR, and we can't validate the entire official build pipeline either.

@ViktorHofer
Copy link
Member Author

ViktorHofer commented Feb 23, 2021 •
edited
Loading

Just to be clear, official build was fine for the previous commit b121bca and failed for c9fe433?

Exactly:

image

Is it different than what we had before? Could you share a screenshot of expected result. This is from before (which PR made more generalized):

From the previous successful official build.

Linux arm:

image

Linux arm musl as it should be:
image

What is interesting is that in the good build for Linux musl arm and Linux musl arm64, I don't see a runtime.native.System.IO.Ports asset published which probably means that it wasn't packaged for those platforms.

I think if 153 legs of public CI which run for > 4 hrs. is not validating something, then it shouldn't be an error. :)

Absolutely, I agree. We need to follow-up on this to make sure that we have protection in place that checks if we have colliding packages in different legs that would later then be uploaded. This has happened numerous times in the past with musl in play...

@ViktorHofer ViktorHofer mentioned this issue Feb 23, 2021
Merged
@ViktorHofer
Copy link
Member Author

Removing blocking-official-build label as the change has been reverted meanwhile.

@ericstj
Copy link
Member

ericstj commented Jun 29, 2021

I think the failure here was caused by $(PackageRID) value being calculated incorrectly. It’s hard to understand why without binlog (too many related properties) but if this were to be brought back that is where to look for the problem.

@am11 am11 mentioned this issue Jun 29, 2021
Merged
@hoyosjs
Copy link
Member

hoyosjs commented Jul 30, 2021

@ViktorHofer is this issue still relevant?

@ViktorHofer
Copy link
Member Author

We reverted the change some months ago and the issue can be closed. Thanks for reminding.

@ghost ghost locked as resolved and limited conversation to collaborators Aug 29, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area-Infrastructure
Projects
None yet
Milestone
6.0.0
Development

No branches or pull requests
4 participants
@ViktorHofer @mmitche @ericstj @hoyosjs