Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mec15xxevb_assy6853: running tests/subsys/power/power_mgmt_soc failed #30192

Closed
maksimmasalski opened this issue Nov 23, 2020 · 9 comments · Fixed by #31765
Closed

mec15xxevb_assy6853: running tests/subsys/power/power_mgmt_soc failed #30192

maksimmasalski opened this issue Nov 23, 2020 · 9 comments · Fixed by #31765
Assignees
Labels
area: Power Management bug The issue is a bug, or the PR is fixing a bug platform: Microchip MEC Microchip MEC Platform priority: low Low impact/importance bug

Comments

@maksimmasalski
Copy link
Collaborator

maksimmasalski commented Nov 23, 2020

Describe the bug

  1. Test test_pm_multithread and test_pm_singlethread fail, Assertion failed in both cases.
  2. Also at the end of test happens kernel error
E: ***** USAGE FAULT ***** 
E:   Illegal use of the EPSR

To Reproduce
Steps to reproduce the behavior:

  1. west build -p -b mec15xxevb_assy6853 tests/subsys/power/power_mgmt_soc
  2. west flash
  3. Open terminal and See error

Expected behavior
Test must pass.

Impact
Power Management issue.

Logs and console output

*** Booting Zephyr OS build zephyr-v2.4.0-1755-g0a0404df29ed  ***
Running test suite test_power_mgmt
===================================================================
START - test_pm_dummyinit
I: PM dummy single-thread test started for one cycle
I: About to enter light sleep
PM >
I: PM sleep entry latency 0.001 seconds
I: Wake from Light Sleep
PM <
I: PM sleep residency 1.502 seconds
I: About to enter deep Sleep
PM >
I: PM sleep entry latency 0.001 seconds
I: Wake from Deep Sleep
PM <
I: PM sleep residency 3.102 seconds
I: PM dummy single-thread completed
 PASS - test_pm_dummyinit
===================================================================
START - test_pm_multithread
I: Thread task A init
I: Thread task B init
I: PM multi-thread test started for cycles: 2
I: Suspend...
I: About to enter light sleep
PM >
I: PM sleep entry latency 0.001 seconds
I: Wake from Light Sleep
PM <
I: PM sleep residency 1.503 seconds
I: Resume
I: Suspend...
I: About to enter deep sleep
PM >
I: PM sleep entry latency 0.000 seconds
I: Wake from Deep Sleep
PM <
I: PM sleep residency 3.103 seconds
I: Resume
I: Suspend...
I: About to enter light sleep
PM >
I: PM sleep entry latency 0.001 seconds
I: Wake from Light Sleep
PM <
I: PM sleep residency 1.502 seconds
I: Resume
I: Suspend...
I: About to enter deep sleep
PM >
I: PM sleep entry latency 0.001 seconds
I: Wake from Deep Sleep
PM <
I: PM sleep residency 3.102 seconds
I: Resume
I: PM multi-thread completed
I: PM state[0] entry counter 2

I: PM state[0] exit counter 2

I: PM state[1] entry counter 2

I: PM state[1] exit counter 1


    Assertion failed at WEST_TOPDIR/zephyr/tests/subsys/power/power_mgmt_soc/sr)
PM counters entry/exit mismatch
 FAIL - test_pm_multithread
===================================================================
START - test_pm_singlethread
I: PM single-thread test started for cycles: 2
I: About to enter light sleep
PM >
I: PM sleep entry latency 0.000 seconds
I: Wake from Light Sleep
PM <
I: PM sleep residency 1.503 seconds
I: About to enter deep Sleep
PM >
I: PM sleep entry latency 0.000 seconds
I: Wake from Deep Sleep
PM <
I: PM sleep residency 3.103 seconds
I: About to enter light sleep
PM >
I: PM sleep entry latency 0.000 seconds
I: Wake from Light Sleep
PM <
I: PM sleep residency 1.503 seconds
I: About to enter deep Sleep
PM >
I: PM sleep entry latency 0.000 seconds
I: Wake from Deep Sleep
PM <
I: PM sleep residency 3.103 seconds
I: PM single-thread completed
I: PM state[0] entry counter 2

I: PM state[0] exit counter 2

I: PM state[1] entry counter 4

I: PM state[1] exit counter 3


    Assertion failed at WEST_TOPDIR/zephyr/tests/subsys/power/power_mgmt_soc/sr)
PM counters entry/exit mismatch
 FAIL - test_pm_singlethread
===================================================================
Test suite test_power_mgmt failed.
E: ***** USAGE FAULT *****
E:   Illegal use of the EPSR
E: r0/a1:  0x00000023  r1/a2:  0x001185a8  r2/a3:  0x001185a8
E: r3/a4:  0x00000001 r12/ip:  0x00000000 r14/lr:  0x000e2703
E:  xpsr:  0x00000000
E: Faulting instruction address (r15/pc): 0x0000000a
E: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
E: Current thread: 0x1185a8 (main)
E: Halting system

Environment (please complete the following information):

  • OS: Linux Fedora 28
  • Toolchain Zephyr SDK 0.11.4
  • Commit SHA 0a0404d
@maksimmasalski maksimmasalski added bug The issue is a bug, or the PR is fixing a bug area: Power Management platform: Microchip MEC Microchip MEC Platform labels Nov 23, 2020
@s-kelvin s-kelvin self-assigned this Nov 25, 2020
@nashif nashif added the priority: medium Medium impact/importance bug label Dec 1, 2020
@s-kelvin
Copy link
Collaborator

s-kelvin commented Dec 2, 2020

The testcase worked well on zephyr version v2.3.0. The failure might be because of the code change in versions.

*** Booting Zephyr OS build zephyr-v2.3.0-1348-g32d6aa724f5c  ***
Running test suite test_power_mgmt
===================================================================
START - test_pm_dummyinit
I: PM dummy single-thread test started for one cycle
I: About to enter light sleep
PM >
I: PM sleep entry latency 0.001 seconds
I: Wake from Light Sleep
PM <
I: PM sleep residency 1.502 seconds
I: About to enter deep Sleep
PM >
I: PM sleep entry latency 0.001 seconds
I: Wake from Deep Sleep
PM <
I: PM sleep residency 3.102 seconds
I: PM dummy single-thread completed
PASS - test_pm_dummyinit
===================================================================
START - test_pm_multithread
I: Thread task A init
I: Thread task B init
I: PM multi-thread test started for cycles: 2
I: Suspend...
I: About to enter light sleep
PM >
I: PM sleep entry latency 0.000 seconds
I: Wake from Light Sleep
PM <
I: PM sleep residency 1.503 seconds
I: Resume
I: Suspend...
I: About to enter deep sleep
PM >
I: PM sleep entry latency 0.001 seconds
I: Wake from Deep Sleep
PM <
I: PM sleep residency 3.102 seconds
I: Resume
I: Suspend...
I: About to enter light sleep
PM >
I: PM sleep entry latency 0.001 seconds
I: Wake from Light Sleep
PM <
I: PM sleep residency 1.503 seconds
I: Resume
I: Suspend...
I: About to enter deep sleep
PM >
I: PM sleep entry latency 0.000 seconds
I: Wake from Deep Sleep
PM <
I: PM sleep residency 3.103 seconds
I: Resume
I: PM multi-thread completed
I: PM state[0] entry counter 2
I: PM state[0] exit counter 2
I: PM state[1] entry counter 2
I: PM state[1] exit counter 2
PASS - test_pm_multithread
===================================================================
START - test_pm_singlethread
I: PM single-thread test started for cycles: 2
I: About to enter light sleep
PM >
I: PM sleep entry latency 0.001 seconds
I: Wake from Light Sleep
PM <
I: PM sleep residency 1.503 seconds
I: About to enter deep Sleep
PM >
I: PM sleep entry latency 0.001 seconds
I: Wake from Deep Sleep
PM <
I: PM sleep residency 3.103 seconds
I: About to enter light sleep
PM >
I: PM sleep entry latency 0.001 seconds
I: Wake from Light Sleep
PM <
I: PM sleep residency 1.503 seconds
I: About to enter deep Sleep
PM >
I: PM sleep entry latency 0.000 seconds
I: Wake from Deep Sleep
PM <
I: PM sleep residency 3.103 seconds
I: PM single-thread completed
I: PM state[0] entry counter 2
I: PM state[0] exit counter 2
I: PM state[1] entry counter 2
I: PM state[1] exit counter 2
PASS - test_pm_singlethread
===================================================================
Test suite test_power_mgmt succeeded
E: ***** USAGE FAULT *****
E:   Illegal use of the EPSR
E: r0/a1:  0x00000000  r1/a2:  0x00118598  r2/a3:  0x00118598
E: r3/a4:  0x00000000 r12/ip:  0x00000000 r14/lr:  0x000e246f
E:  xpsr:  0x60000000
E: Faulting instruction address (r15/pc): 0x00000014
E: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
E: Current thread: 0x00118598 (main)
E: Halting system

@maksimmasalski
Copy link
Collaborator Author

The testcase worked well on zephyr version v2.3.0. The failure might be because of the code change in versions.

*** Booting Zephyr OS build zephyr-v2.3.0-1348-g32d6aa724f5c  ***
Running test suite test_power_mgmt
===================================================================
START - test_pm_dummyinit
I: PM dummy single-thread test started for one cycle
I: About to enter light sleep
PM >
I: PM sleep entry latency 0.001 seconds
I: Wake from Light Sleep
PM <
I: PM sleep residency 1.502 seconds
I: About to enter deep Sleep
PM >
I: PM sleep entry latency 0.001 seconds
I: Wake from Deep Sleep
PM <
I: PM sleep residency 3.102 seconds
I: PM dummy single-thread completed
PASS - test_pm_dummyinit
===================================================================
START - test_pm_multithread
I: Thread task A init
I: Thread task B init
I: PM multi-thread test started for cycles: 2
I: Suspend...
I: About to enter light sleep
PM >
I: PM sleep entry latency 0.000 seconds
I: Wake from Light Sleep
PM <
I: PM sleep residency 1.503 seconds
I: Resume
I: Suspend...
I: About to enter deep sleep
PM >
I: PM sleep entry latency 0.001 seconds
I: Wake from Deep Sleep
PM <
I: PM sleep residency 3.102 seconds
I: Resume
I: Suspend...
I: About to enter light sleep
PM >
I: PM sleep entry latency 0.001 seconds
I: Wake from Light Sleep
PM <
I: PM sleep residency 1.503 seconds
I: Resume
I: Suspend...
I: About to enter deep sleep
PM >
I: PM sleep entry latency 0.000 seconds
I: Wake from Deep Sleep
PM <
I: PM sleep residency 3.103 seconds
I: Resume
I: PM multi-thread completed
I: PM state[0] entry counter 2
I: PM state[0] exit counter 2
I: PM state[1] entry counter 2
I: PM state[1] exit counter 2
PASS - test_pm_multithread
===================================================================
START - test_pm_singlethread
I: PM single-thread test started for cycles: 2
I: About to enter light sleep
PM >
I: PM sleep entry latency 0.001 seconds
I: Wake from Light Sleep
PM <
I: PM sleep residency 1.503 seconds
I: About to enter deep Sleep
PM >
I: PM sleep entry latency 0.001 seconds
I: Wake from Deep Sleep
PM <
I: PM sleep residency 3.103 seconds
I: About to enter light sleep
PM >
I: PM sleep entry latency 0.001 seconds
I: Wake from Light Sleep
PM <
I: PM sleep residency 1.503 seconds
I: About to enter deep Sleep
PM >
I: PM sleep entry latency 0.000 seconds
I: Wake from Deep Sleep
PM <
I: PM sleep residency 3.103 seconds
I: PM single-thread completed
I: PM state[0] entry counter 2
I: PM state[0] exit counter 2
I: PM state[1] entry counter 2
I: PM state[1] exit counter 2
PASS - test_pm_singlethread
===================================================================
Test suite test_power_mgmt succeeded
E: ***** USAGE FAULT *****
E:   Illegal use of the EPSR
E: r0/a1:  0x00000000  r1/a2:  0x00118598  r2/a3:  0x00118598
E: r3/a4:  0x00000000 r12/ip:  0x00000000 r14/lr:  0x000e246f
E:  xpsr:  0x60000000
E: Faulting instruction address (r15/pc): 0x00000014
E: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
E: Current thread: 0x00118598 (main)
E: Halting system

Yes, it might be a code change. You have to find the latest stable zephyr commit which worked well, and then investigate what code changes where did right after that causing an error.

@s-kelvin
Copy link
Collaborator

s-kelvin commented Dec 3, 2020

The testcase passed without failures by cherry-pick the testcase on top of the commit of the release of v2.4.0 on Sep.26. Then I used bisect between the cherry-pick commit and my merged commit #27733, which gave me this commit as the exact one that introduced the change causing the failure of this testcase. @andrewboie Can you look into this please?

@s-kelvin s-kelvin assigned andrewboie and unassigned s-kelvin Dec 3, 2020
@rljordan rljordan assigned ceolin and unassigned andrewboie Dec 16, 2020
ceolin pushed a commit to ceolin/zephyr that referenced this issue Jan 28, 2021
Test suite test_power_mgmt succeeded
E: ***** USAGE FAULT *****
E:   Illegal use of the EPSR
E: r0/a1:  0x00000000  r1/a2:  0x00118598  r2/a3:  0x00118598
E: r3/a4:  0x00000000 r12/ip:  0x00000000 r14/lr:  0x000e246f
E:  xpsr:  0x60000000
E: Faulting instruction address (r15/pc): 0x00000014
E: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
E: Current thread: 0x00118598 (main)
E: Halting system

Fixes: zephyrproject-rtos#30192

Signed-off-by: Flavio Ceolin <[email protected]>
@ceolin ceolin added the has-pr label Jan 28, 2021
nashif pushed a commit that referenced this issue Jan 29, 2021
Test suite test_power_mgmt succeeded
E: ***** USAGE FAULT *****
E:   Illegal use of the EPSR
E: r0/a1:  0x00000000  r1/a2:  0x00118598  r2/a3:  0x00118598
E: r3/a4:  0x00000000 r12/ip:  0x00000000 r14/lr:  0x000e246f
E:  xpsr:  0x60000000
E: Faulting instruction address (r15/pc): 0x00000014
E: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
E: Current thread: 0x00118598 (main)
E: Halting system

Fixes: #30192

Signed-off-by: Flavio Ceolin <[email protected]>
@s-kelvin
Copy link
Collaborator

s-kelvin commented Feb 1, 2021

This has not been fixed. See the latest log on micro chip below:

*** Booting Zephyr OS build v2.5.0-rc1-133-gdfd8e7522d11  ***
Running test suite test_power_mgmt
===================================================================
START - test_pm_dummyinit
I: PM dummy single-thread test started for one cycle
I: About to enter light sleep
PM >
I: Wake from Light Sleep
PM <
I: PM sleep residency 1.503 seconds
I: About to enter deep Sleep
PM >
I: Wake from Deep Sleep
PM <
I: PM sleep residency 3.103 seconds
I: PM dummy single-thread completed
PASS - test_pm_dummyinit
===================================================================
START - test_pm_multithread
I: Thread task A init
I: Thread task B init
I: PM multi-thread test started for cycles: 2
I: Suspend...
I: About to enter light sleep
PM >
I: PM sleep entry latency 0.001 seconds
I: Wake from Light Sleep
PM <
I: PM sleep residency 1.502 seconds
I: Resume
I: Suspend...
I: About to enter deep sleep
PM >
I: PM sleep entry latency 0.000 seconds
I: Wake from Deep Sleep
PM <
I: PM sleep residency 3.103 seconds
I: Resume
I: Suspend...
I: About to enter light sleep
PM >
I: PM sleep entry latency 0.000 seconds
I: Wake from Light Sleep
PM <
I: PM sleep residency 1.503 seconds
I: Resume
I: Suspend...
I: About to enter deep sleep
PM >
I: PM sleep entry latency 0.001 seconds
I: Wake from Deep Sleep
PM <
I: PM sleep residency 3.102 seconds
I: Resume
I: PM multi-thread completed
I: PM state[0] entry counter 2
I: PM state[0] exit counter 2
I: PM state[1] entry counter 2
I: PM state[1] exit counter 1
Assertion failed at WEST_TOPDIR/zephyr/tests/subsys/power/power_mgmt_soc/src/power_mgmt.c:117: pm_check_counters: (pm_counters[i].entry_cnt not equal to pm_counters[i].exit_cnt)
PM counters entry/exit mismatch
E: ***** USAGE FAULT *****
E:   Illegal load of EXC_RETURN into PC
E: r0/a1:  0x00000000  r1/a2:  0xe000ed00  r2/a3:  0x00118dd0
E: r3/a4:  0x00000000 r12/ip:  0x0000000a r14/lr:  0x000e2abd
E:  xpsr:  0x00119c00
E: Faulting instruction address (r15/pc): 0x00119cb8
E: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
E: Current thread: 0x1185b8 (main)
E: Halting system

@s-kelvin s-kelvin reopened this Feb 1, 2021
@ceolin
Copy link
Member

ceolin commented Feb 2, 2021

That is not the same error message, I'd used cc1352r1_launchxl to fix the first report that was consistently happening there as well. But this one I'm not being able to reproduce. It seems though that the stack is being corrupted, would mind to test in this board with the following option:
CONFIG_IDLE_STACK_SIZE=2048

@nashif nashif added priority: low Low impact/importance bug and removed priority: medium Medium impact/importance bug labels Feb 8, 2021
@carlescufi
Copy link
Member

would mind to test in this board with the following option:
CONFIG_IDLE_STACK_SIZE=2048

@maksimmasalski could you re-run the test with that option to see if you get rid of the fault?

@s-kelvin
Copy link
Collaborator

s-kelvin commented Feb 18, 2021

Here's the log after changing idle stack size to 2048. It just seems it's not a once-for-all solution.

Build log
*** Booting Zephyr OS build zephyr-v2.5.0-197-gca5272f239cd  ***
Running test suite test_power_mgmt
===================================================================
START - test_pm_dummyinit
I: PM dummy single-thread test started for one cycle
I: About to enter light sleep
PM >
I: Wake from Light Sleep
PM <
I: PM sleep residency 1.503 seconds
I: About to enter deep Sleep
PM >
I: Wake from Deep Sleep
PM <
I: PM sleep residency 3.103 seconds
I: PM dummy single-thread completed
PASS - test_pm_dummyinit
===================================================================
START - test_pm_multithread
I: Thread task A init
I: Thread task B init
I: PM multi-thread test started for cycles: 2
I: Suspend...
I: About to enter light sleep
PM >
I: PM sleep entry latency 0.001 seconds
I: Wake from Light Sleep
PM <
I: PM sleep residency 1.503 seconds
I: Resume
I: Suspend...
I: About to enter deep sleep
PM >
I: PM sleep entry latency 0.000 seconds
*** Booting Zephyr OS build zephyr-v2.5.0-197-gca5272f239cd  ***
Running test suite test_power_mgmt
===================================================================
START - test_pm_dummyinit
I: PM dummy single-thread test started for one cycle
I: About to enter light sleep
PM >
I: Wake from Light Sleep
PM <
I: PM sleep residency 1.503 seconds
I: About to enter deep Sleep
PM >
I: Wake from Deep Sleep
PM <
I: PM sleep residency 3.103 seconds
I: PM dummy single-thread completed
PASS - test_pm_dummyinit
===================================================================
START - test_pm_multithread
I: Thread task A init
I: Thread task B init
I: PM multi-thread test started for cycles: 2
I: Suspend...
I: About to enter light sleep
PM >
I: PM sleep entry latency 0.001 seconds
I: Wake from Light Sleep

...skipping 1 line
I: PM sleep residency 1.503 seconds
I: Resume
I: Suspend...
I: About to enter deep sleep
PM >
I: PM sleep entry latency 0.000 seconds
I: Wake from Deep Sleep
PM <
I: PM sleep residency 3.103 seconds
I: Resume
I: Suspend...
I: About to enter light sleep
PM >
I: PM sleep entry latency 0.001 seconds
I: Wake from Light Sleep
PM <
I: PM sleep residency 1.502 seconds
I: Resume
I: Suspend...
I: About to enter deep sleep
PM >
I: PM sleep entry latency 0.001 seconds
I: Wake from Deep Sleep
PM <
I: PM sleep residency 3.102 seconds
I: Resume
I: PM multi-thread completed
I: PM state[0] entry counter 2
I: PM state[0] exit counter 2
I: PM state[1] entry counter 2
I: PM state[1] exit counter 1
Assertion failed at WEST_TOPDIR/zephyr/tests/subsys/power/power_mgmt_soc/src/pow
er_mgmt.c:117: pm_check_counters: (pm_counters[i].entry_cnt not equal to pm_coun
ters[i].exit_cnt)
PM counters entry/exit mismatch
FAIL - test_pm_multithread
===================================================================
START - test_pm_singlethread
I: PM single-thread test started for cycles: 2
I: About to enter light sleep
PM >
I: PM sleep entry latency 0.000 seconds
I: Wake from Light Sleep
PM <
I: PM sleep residency 1.503 seconds
I: About to enter deep Sleep
PM >
I: PM sleep entry latency 0.000 seconds
I: Wake from Deep Sleep
PM <
I: PM sleep residency 3.103 seconds
I: About to enter light sleep
PM >
I: PM sleep entry latency 0.000 seconds
I: Wake from Light Sleep
PM <
I: PM sleep residency 1.503 seconds
I: About to enter deep Sleep
PM >
I: PM sleep entry latency 0.000 seconds
I: Wake from Deep Sleep

...skipping 1 line
I: PM sleep residency 3.103 seconds
I: PM single-thread completed
I: PM state[0] entry counter 2
I: PM state[0] exit counter 2
I: PM state[1] entry counter 4
I: PM state[1] exit counter 2
Assertion failed at WEST_TOPDIR/zephyr/tests/subsys/power/power_mgmt_soc/src/pow
er_mgmt.c:117: pm_check_counters: (pm_counters[i].entry_cnt not equal to pm_coun
ters[i].exit_cnt)
PM counters entry/exit mismatch
FAIL - test_pm_singlethread
===================================================================
Test suite test_power_mgmt failed.
===================================================================
PROJECT EXECUTION FAILED

@ceolin
Copy link
Member

ceolin commented Feb 22, 2021

that is a different problem, let me look it. This one seems to be a real issue and not only a stack overflow

@s-kelvin
Copy link
Collaborator

s-kelvin commented Mar 1, 2021

This is no longer a failure since 0ba9437.

@s-kelvin s-kelvin closed this as completed Mar 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: Power Management bug The issue is a bug, or the PR is fixing a bug platform: Microchip MEC Microchip MEC Platform priority: low Low impact/importance bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants