[android] Failing E2E Android - Devices Helix Tests Build_Debug tests

[ivanpovazan]

Build

https://dev.azure.com/dnceng-public/public/_build/results?buildId=750141

Build leg reported

E2E Android - Devices Helix Tests Build_Debug failed

Pull Request

dotnet/xharness#1233

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

{
  "ErrorMessage": "crit: Install failure: Test command cannot continue",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=750141
Error message validated: [crit: Install failure: Test command cannot continue]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 7/29/2024 6:26:51 PM UTC

Report

Build	Definition	Test	Pull Request
839397	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution
839394	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution
839382	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution
838677	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution	dotnet/xharness#1295
838368	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution	dotnet/xharness#1292
837857	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution	dotnet/xharness#1291
837837	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution	dotnet/xharness#1291
836737	dotnet/arcade	System.Buffers.Tests-arm64-v8a-custom-command.WorkItemExecution
836584	dotnet/arcade	System.Buffers.Tests-arm64-v8a.WorkItemExecution	dotnet/arcade#15153
833645	dotnet/runtime	System.Threading.Tasks.Dataflow.Tests.WorkItemExecution
832523	dotnet/arcade	System.Buffers.Tests-arm64-v8a.WorkItemExecution	dotnet/arcade#15146
832047	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution	dotnet/xharness#1288
832010	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution
831971	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution	dotnet/xharness#1288
825305	dotnet/runtime	System.Threading.ThreadPool.Tests.WorkItemExecution
823678	dotnet/arcade	System.Buffers.Tests-arm64-v8a.WorkItemExecution
823581	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution
822315	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution
822026	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution	dotnet/xharness#1282
822072	dotnet/arcade	System.Buffers.Tests-arm64-v8a-custom-command.WorkItemExecution	dotnet/arcade#15087
819558	dotnet/runtime	System.Security.Cryptography.OpenSsl.Tests.WorkItemExecution
819157	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution	dotnet/xharness#1279
819143	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution
819033	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution	dotnet/xharness#1277
818848	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution	dotnet/xharness#1277
818803	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution	dotnet/xharness#1277
815614	dotnet/arcade	System.Buffers.Tests-arm64-v8a-custom-command.WorkItemExecution	dotnet/arcade#15087
817968	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution	dotnet/xharness#1277
815834	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution
815712	dotnet/xharness	System.Buffers.Tests-x86.WorkItemExecution	dotnet/xharness#1276
812840	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution
810489	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution
810474	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution
809758	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution	dotnet/xharness#1273
809552	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution	dotnet/xharness#1272
809474	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution	dotnet/xharness#1271
809392	dotnet/arcade	System.Buffers.Tests-arm64-v8a.WorkItemExecution	dotnet/arcade#15087
809353	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution	dotnet/xharness#1270
809244	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution
809163	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution	dotnet/xharness#1268
808489	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution
808108	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution	dotnet/xharness#1268
808047	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution	dotnet/xharness#1267
807953	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution	dotnet/xharness#1267
807915	dotnet/arcade	System.Buffers.Tests-arm64-v8a-custom-command.WorkItemExecution
807376	dotnet/xharness	System.Buffers.Tests-x86.WorkItemExecution
806990	dotnet/arcade	System.Buffers.Tests-arm64-v8a-custom-command.WorkItemExecution	dotnet/arcade#15020
806807	dotnet/xharness	System.Buffers.Tests-arm64-v8a.WorkItemExecution	dotnet/xharness#1266
806688	dotnet/arcade	System.Buffers.Tests-arm64-v8a-custom-command.WorkItemExecution

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	14	49

[ivanpovazan]

Problem

This issue seems to be occurring more and more, and xharness CI is mostly red.

The problem comes from the fact that in some jobs we are testing debug apks on physical Android devices, which gives:

adb: failed to install D:\h\w\A8C809B3\w\A92C0917\e\System.Buffers.Tests-arm64-v8a.apk: Failure [INSTALL_FAILED_VERIFICATION_FAILURE: Package Verification Result]
             
[02:24:27] crit: Install failure: Test command cannot continue

(link to the latest failure on DNCENGWIN-007 : https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-xharness-refs-heads-release-90-68d2120ee0c9454f80/System.Buffers.Tests-arm64-v8a.Attempt.3/1/console.03621459.log?helixlogtype=result

Installing such apks can be rejected by the device, if Verify apps over USB setting is enabled on it.

Assumption

My assumption is that the tethered devices were initially configured to have this verification disabled in device settings - allowing debug apks to be installed and tested, and that a factory reset happened recently, which caused switching this setting on again and caused the failures we are now seeing.

Ask

DNCENGWIN-070 had a successful run https://dev.azure.com/dnceng-public/public/_build/results?buildId=811051&view=logs&j=00f02412-5f5e-5b12-0780-d2e57de065ba&t=b1200827-0fdc-54c0-1776-c7b8da453b26
while DNCENGWIN-007 seems to fail constantly. Checking the Verify apps over USB setting on tethered devices on these machines would help us figure out the next steps.

@ilyas1974, could someone check this out?

[ilyas1974]

Hi Ivan, I have taken DNCENG-007 out of rotation so no additional jobs will use that system while we investigate the failure you mentioned above. I have created https://portal.microsofticm.com/imp/v5/incidents/details/546521338 to have the system in question looked at.

[ivanpovazan]

Thank you!
Should I add some context on the created incident ticket?