Stopping agent: Faulting module name sechost.dll error

[Rebits]

Wazuh version	Component	Install type	Install method	Platform
4.4.0-40400.20220406	Agent	Agent	Packages	Windows Server 2019

Description

It has been detected in the nightly wazuh/wazuh-qa#2769, that AR tier 0 tests fail in Windows agent
The restart wazuh command produces an unexpected error in the agent.

Steps to reproduce

• Deploy a Windows agent and register to a manager
• Add the following custom rules/decoders:

<rule id="111009" level="9">
  <program_name>example</program_name>
  <description>User logged</description>
</rule>

<decoder name="example">
  <program_name>^example</program_name>
</decoder>

<decoder name="example">
  <parent>example</parent>
  <regex>User '(\w+)' logged from '(\d+.\d+.\d+.\d+)'</regex>
  <order>user, srcip</order>
</decoder>

• Activate active response in the manager configuration

<active-response>
  <command>restart-wazuh</command>
  <location>local</location>
  <rules_id>111009</rules_id>
</active-response>

• Restart the wazuh-manager

• Configure Syslog file monitoring in the agent configuration

  <localfile>
    <location>c:\example.log</location>
    <log_format>syslog</log_format>
  </localfile>

• Add the following line to the example.log file

Dec 25 20:45:02 MyHost example[12345]: User 'admin' logged from '192.168.1.100'

• Restart the Wazuh agent
• Check that an unexpected error occurs in the Windows agent and that it will not restart.

2022 Apr 07 10:08:47 (windows) any->EventChannel
Rule: 60602 (level 9) -> 'Windows application error event.'
{"win":{"system":{"providerName":"Application Error","eventID":"1000","level":"2","task":"100","keywords":"0x80000000000000","systemTime":"2022-04-07T17:08:47.057399700Z","eventRecordID":"1577","channel":"Application","computer":"windows","severityValue":"ERROR","message":"\"Faulting application name: wazuh-agent.exe, version: 0.0.0.0, time stamp: 0x624b731f\r\nFaulting module name: sechost.dll, version: 10.0.17763.1, time stamp: 0xec52cb01\r\nException code: 0xc0000005\r\nFault offset: 0x00015896\r\nFaulting process id: 0xfd0\r\nFaulting application start time: 0x01d84aa1ff944f83\r\nFaulting application path: C:\\Program Files (x86)\\ossec-agent\\wazuh-agent.exe\r\nFaulting module path: C:\\Windows\\System32\\sechost.dll\r\nReport Id: dc1d310b-c2b6-41c5-b77a-379740cdcec3\r\nFaulting package full name: \r\nFaulting package-relative application ID: \""},"eventdata":{"data":"wazuh-agent.exe, 0.0.0.0, 624b731f, sechost.dll, 10.0.17763.1, ec52cb01, c0000005, 00015896, fd0, 01d84aa1ff944f83, C:\\\\Program Files (x86)\\\\ossec-agent\\\\wazuh-agent.exe, C:\\\\Windows\\\\System32\\\\sechost.dll, dc1d310b-c2b6-41c5-b77a-379740cdcec3"}}}
win.system.providerName: Application Error
win.system.eventID: 1000
win.system.level: 2
win.system.task: 100
win.system.keywords: 0x80000000000000
win.system.systemTime: 2022-04-07T17:08:47.057399700Z
win.system.eventRecordID: 1577
win.system.channel: Application
win.system.computer: windows
win.system.severityValue: ERROR
win.system.message: "Faulting application name: wazuh-agent.exe, version: 0.0.0.0, time stamp: 0x624b731f
Faulting module name: sechost.dll, version: 10.0.17763.1, time stamp: 0xec52cb01
Exception code: 0xc0000005
Fault offset: 0x00015896
Faulting process id: 0xfd0
Faulting application start time: 0x01d84aa1ff944f83
Faulting application path: C:\Program Files (x86)\ossec-agent\wazuh-agent.exe
Faulting module path: C:\Windows\System32\sechost.dll
Report Id: dc1d310b-c2b6-41c5-b77a-379740cdcec3
Faulting package full name: 
Faulting package-relative application ID: "
win.eventdata.data: wazuh-agent.exe, 0.0.0.0, 624b731f, sechost.dll, 10.0.17763.1, ec52cb01, c0000005, 00015896, fd0, 01d84aa1ff944f83, C:\\Program Files (x86)\\ossec-agent\\wazuh-agent.exe, C:\\Windows\\System32\\sechost.dll, dc1d310b-c2b6-41c5-b77a-379740cdcec3

[Deblintrake09]

During research for Wazuh-qa Issues #2769 and #3057, it was found that everytime agent was stopped, the event in the previous comment would be generated. The following is a snapshot of the Windows Event Viewer showing all instances of the error caused by stopping the agent with the test_active_responte/test_execd_restart.py test module.

Event example:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
 <System>
  <Provider Name="Application Error" /> 
  <EventID Qualifiers="0">1000</EventID> 
  <Level>2</Level> 
  <Task>100</Task> 
  <Keywords>0x80000000000000</Keywords> 
  <TimeCreated SystemTime="2022-08-01T11:53:10.951288500Z" /> 
  <EventRecordID>12923</EventRecordID> 
  <Channel>Application</Channel> 
  <Computer>EC2AMAZ-NRK2S46</Computer> 
  <Security /> 
  </System>
 <EventData>
  <Data>wazuh-agent.exe</Data> 
  <Data>0.0.0.0</Data> 
  <Data>62e04b69</Data> 
  <Data>sechost.dll</Data> 
  <Data>10.0.14393.2515</Data> 
  <Data>5b88518d</Data> 
  <Data>c0000005</Data> 
  <Data>0000bd56</Data> 
  <Data>12e8</Data> 
  <Data>01d8a59d45c00449</Data> 
  <Data>C:\Program Files (x86)\ossec-agent\wazuh-agent.exe</Data> 
  <Data>C:\Windows\System32\sechost.dll</Data> 
  <Data>a9fa1e64-894b-4932-b3b5-caa9ae514233</Data> 
  <Data /> 
  <Data /> 
  </EventData>
  </Event>

Further Research

• New machine with no agent restarts. No application error.
  
• Restarted from agent's UI - Application error appears
  
• Stop Agent from agent's UI - Application error appears
  

[Dwordcito]

RCA:
The root cause of this issue, is the misuse of __stdcall(AKA WINAPI) calling convention.

because of the function pointer it is treating a function as cdecl, calling a function that has stdcall convention

wazuh_agent!stop_wmodules:
00902030 53              push    ebx
00902031 83ec18          sub     esp,18h
00902034 8b1dd06ba300    mov     ebx,dword ptr [wazuh_agent!wmodules (00a36bd0)]
0090203a 85db            test    ebx,ebx
0090203c 741e            je      wazuh_agent!stop_wmodules+0x2c (0090205c)
0090203e 6690            nop
00902040 8b4304          mov     eax,dword ptr [ebx+4]
00902043 8b4014          mov     eax,dword ptr [eax+14h]
00902046 85c0            test    eax,eax
00902048 740b            je      wazuh_agent!stop_wmodules+0x25 (00902055)
0090204a 8b530c          mov     edx,dword ptr [ebx+0Ch]
0090204d 891424          mov     dword ptr [esp],edx
00902050 ffd0            call    eax
**00902052 83ec04          sub     esp,4**
00902055 8b5b10          mov     ebx,dword ptr [ebx+10h]
00902058 85db            test    ebx,ebx
0090205a 75e4            jne     wazuh_agent!stop_wmodules+0x10 (00902040)
0090205c 83c418          add     esp,18h
0090205f 5b              pop     ebx
00902060 c3              ret
00902061 8db42600000000  lea     esi,[esi]
00902068 8db42600000000  lea     esi,[esi]
0090206f 90

as you can see in this address (00902052), post to the call of a function you want to do the cleanup on the stack, moving the base pointer, this is typical of cdecl, when the above function is executed as stdcall, which already does this cleanup .

Callee function on this case -> wazuh_agent!wm_sys_stop