WPK Update on ALAS 2022 #3415

Deblintrake09 · 2022-10-04T15:46:30Z

Target version	Related issue	Related PR
4.4.0	wazuh/wazuh#14982	wazuh/wazuh#15151

Description

This development fixes WPK upgrade process for ALAS2022 agents. Testing needs to be done in order to verify it works properly.

Proposed checks

Update with WPK built from Sources

Steps to reproduce

Install the 4.3.8-1 agent
Install 4.4.0 Manager
Build WPK (from sources/Jenkins)
Update agent with WPK file
Check agent has been properly updated
Check agent is working

Expected results

Agent upgrades to 4.4.0 and works properly.

chemamartinez · 2022-10-13T15:54:07Z

@Deblintrake09 finally the issue was related to upgrading any agent with aarch64 architecture. Doesn't matter the OS.

Deblintrake09 · 2022-10-17T19:04:30Z

Review data

Tester	PR commit
@Deblintrake09	wazuh/wazuh@`d285b84`

Testing environment

OS	OS version	Deployment	Image/AMI
Centos	8	Remote \| Deployer	ami-029496e60f56b4b13-0
Amazon Linux	2022	`Remote \| AWS`	ami-0f57398c09c46c310

Tested packages

OS	Package
Centos	Manager 4.3.9-1 Manager 4.4.0 - fixed branch
Amazon Linux	Agent 4.3.8-1 Agent 4.3.9-1

Conclusion 🟡

Everything works as expected after the fix however there was an Issue found with the steps taken for the wpk package generation and the agent upgrade through CLI:

The CLI application upgrade_agent would not finish and release console control, even though the agent was updated and restarted. Opened Issue Agent_upgrade does not finish after sucessful upgrade wazuh#15237.
- The Issue was a known error in the documentation, and is fixed by Fix WPK manual generation wazuh-documentation#5701

Status

Deblintrake09 · 2022-10-17T19:14:07Z

Testing results

Work on unfixed Branch 4.3.9-1

Build WPK and upgrade agent 🔴

In Manager

Install Manager 4.3.9-1

  Running scriptlet: wazuh-manager-4.3.9-1.x86_64                                1/1 
  Verificando         : wazuh-manager-4.3.9-1.x86_64                                1/1 

Installed:
  wazuh-manager-4.3.9-1.x86_64

In Agent
2. Install agent 4.3.8

  Running scriptlet: wazuh-agent-4.3.8-1.aarch64                                                                                1/1 
  Verifying        : wazuh-agent-4.3.8-1.aarch64                                                                                1/1 

Installed:
  wazuh-agent-4.3.8-1.aarch64                                                                                                       

Complete!

Install dependencies
Clone Repository

# git clone https://github.com/wazuh/wazuh
# cd wazuh/src
# git checkout fac4fd56911049804c3f1685e6bdcc4e22e926d3

Modify preconfigured vars and uncomment the following lines

# nano etc/preloaded-vars.conf

USER_LANGUAGE="en"
USER_NO_STOP="y"
USER_UPDATE="y"
USER_BINARYINSTALL="y"

Build Agent from sources

    LDFLAGS           '-Wl,-rpath,/../lib' -pthread -lrt -ldl -O2 -Lshared_modules/dbsync/build/lib -Lshared_modules/rsync/build/lib  -Lwazuh_modules/syscollector/build/lib -Ldata_provider/build/lib
    LIBS              -lrt -ldl -lm 
    CC                cc
    MAKE              make
make[1]: se sale del directorio '/home/qa/wazuh/src'

Done building agent

Generate certificates

cd /tmp
openssl req -x509 -new -nodes -newkey rsa:2048 -keyout wpk_root.key -out wpk_root.pem -batch
openssl req -new -nodes -newkey rsa:2048 -keyout wpkcert.key -out wpkcert.csr -subj '/C=US/ST=CA/O=Wazuh'
openssl x509 -req -days 365 -in wpkcert.csr -CA wpk_root.pem -CAkey wpk_root.key -out wpkcert.pem -CAcreateserial
cd -

Delete Unnecesary files

cd ../
rm -rf doc wodles/oscap/content/* gen_ossec.sh add_localfiles.sh Jenkinsfile*
rm -rf src/{addagent,analysisd,client-agent,config,error_messages,external/*,headers,logcollector,monitord,os_auth,os_crypto,os_csyslogd,os_dbdos_execd}
rm -rf src/{os_integrator,os_maild,os_netos_regex,os_xml,os_zlib,remoted,reportd,shared,syscheckd,tests,update,wazuh_db,wazuh_modules}
rm -rf src/win32
rm -rf src/*.a
rm -rf etc/{decoders,lists,rules}
find etc/templates/* -maxdepth 0 -not -name "en" | xargs rm -rf

Build wpk package

tools/agent-upgrade/wpkpack.py /tmp/agent439.wpk /tmp/wpkcert.pem /tmp/wpkcert.key *

Copy wpk file to manager

scp -i KEY-PATH /tmp/agent439.wpk user@ip:/var/ossec/etc

Check that agent is registered

# /var/ossec/bin/agent_control -l

Wazuh agent_control. List of available agents:
   ID: 000, Name: ip-172-31-5-9.ec2.internal (server), IP: 127.0.0.1, Active/Local
   ID: 001, Name: ip-172-31-0-169.ec2.internal, IP: any, Active

Upgrade agent
In the Manager

# /var/ossec/bin/agent_upgrade -a 001 -f /home/qa/agent439.wpk
Agent information not found in database
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/var/ossec/framework/python/lib/python3.9/concurrent/futures/process.py", line 243, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/var/ossec/framework/python/lib/python3.9/site-packages/wazuh-4.3.9-py3.9.egg/wazuh/core/cluster/dapi/dapi.py", line 243, in run_local
    data = f(**f_kwargs)
  File "/var/ossec/framework/python/lib/python3.9/site-packages/wazuh-4.3.9-py3.9.egg/wazuh/rbac/decorators.py", line 420, in wrapper
    result = func(*args, **kwargs) if not skip_execution else None
  File "/var/ossec/framework/python/lib/python3.9/site-packages/wazuh-4.3.9-py3.9.egg/wazuh/agent.py", line 981, in upgrade_agents
    raise WazuhInternalError(error_code, cmd_error=True, extra_message=agent_result['message'])
wazuh.core.exception.WazuhInternalError: Error 1816 - Agent information not found in database
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/ossec/framework/python/lib/python3.9/site-packages/wazuh-4.3.9-py3.9.egg/wazuh/core/cluster/dapi/dapi.py", line 288, in execute_local_request
    data = await asyncio.wait_for(task, timeout=timeout)
  File "/var/ossec/framework/python/lib/python3.9/asyncio/tasks.py", line 442, in wait_for
    return await fut
wazuh.core.exception.WazuhInternalError: Error 1816 - Agent information not found in database
Internal error: 
Traceback (most recent call last):
  File "/var/ossec/framework/scripts/agent_upgrade.py", line 220, in <module>
    main()
  File "/var/ossec/framework/scripts/agent_upgrade.py", line 186, in main
    result = send_command(function=upgrade_agents, command=create_command())
  File "/var/ossec/framework/scripts/agent_upgrade.py", line 115, in send_command
    return raise_if_exc(pool.submit(run, dapi.distribute_function()).result())
  File "/var/ossec/framework/python/lib/python3.9/site-packages/api-4.3.9-py3.9.egg/api/util.py", line 293, in raise_if_exc
    _create_problem(obj)
  File "/var/ossec/framework/python/lib/python3.9/site-packages/api-4.3.9-py3.9.egg/api/util.py", line 263, in _create_problem
    raise ProblemException(status=500 if not code else code, type=exc.type, title=exc.title, detail=exc.message,
connexion.exceptions.ProblemException

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/ossec/framework/scripts/agent_upgrade.py", line 227, in <module>
    if args.debug:
AttributeError: 'Namespace' object has no attribute 'debug'

Work on Fixed Branch

Build WPK and upgrade agent 🟡

**In the manager **

. Install Manager

  Ejecutando scriptlet: wazuh-manager-4.4.0-3415.aarch64.wpk.   1/1 
  Verificando         : wazuh-manager-4.4.0-3415.aarch64.wpk.   1/1 

Instalado:
  wazuh-manager-4.4.0-3415.aarch64.wpk.fix.x86_64

**In Agent **
2. Install agent 4.3.9-1

# /var/ossec/bin/wazuh-control info
WAZUH_VERSION="v4.3.9"
WAZUH_REVISION="40322"
WAZUH_TYPE="agent"

Install dependencies
Clone repo and modify the etc/preloaded-vars.conf file uncommenting the following lines:

# git clone https://github.com/wazuh/wazuh
# cd wazuh/src
# git checkout 14982-fix-aarch64-not-found
# nano etc/preloaded-vars.conf

USER_LANGUAGE="en"
USER_NO_STOP="y"
USER_UPDATE="y"
USER_BINARYINSTALL="y"

Build agent from sources

# make deps TARGET=agent && make TARGET=agent
    -DOSSECHIDS -DUSER="wazuh" -DGROUPGLOBAL="wazuh" -DLinux -DINOTIFY_ENABLED -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DENABLE_SYSC -DENABLE_CISCAT -DENABLE_AUDIT -DCLIENT
Compiler:
    CFLAGS            -pthread -Iexternal/libdb/build_unix/ -Iexternal/pacman/lib/libalpm/ -Iexternal/libarchive/libarchive -Wl,--start-group -Iexternal/audit-userspace/lib -DNDEBUG -O2 -DOSSECHIDS -DUSER="wazuh" -DGROUPGLOBAL="wazuh" -DLinux -DINOTIFY_ENABLED -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DENABLE_SYSC -DENABLE_CISCAT -DENABLE_AUDIT -DCLIENT -pipe -Wall -Wextra -std=gnu99 -I./ -I./headers/ -Iexternal/openssl/include -Iexternal/cJSON/ -Iexternal/libyaml/include -Iexternal/curl/include -Iexternal/msgpack/include -Iexternal/bzip2/ -Ishared_modules/common -Ishared_modules/dbsync/include -Ishared_modules/rsync/include -Iwazuh_modules/syscollector/include  -Idata_provider/include  -Iexternal/libpcre2/include -Iexternal/rpm//builddir/output/include 
    LDFLAGS           '-Wl,-rpath,/../lib' -pthread -lrt -ldl -O2 -Lshared_modules/dbsync/build/lib -Lshared_modules/rsync/build/lib  -Lwazuh_modules/syscollector/build/lib -Ldata_provider/build/lib
    LIBS              -lrt -ldl -lm 
    CC                cc
    MAKE              make
make[1]: Leaving directory '/home/ec2-user/wazuh/src'

Done building agent

Generate certificate WPK to sign wpk and replace root certificate

cd /tmp
openssl req -x509 -new -nodes -newkey rsa:2048 -keyout wpk_root.key -out wpk_root.pem -batch
openssl req -new -nodes -newkey rsa:2048 -keyout wpkcert.key -out wpkcert.csr -subj '/C=US/ST=CA/O=Wazuh'
openssl x509 -req -days 365 -in wpkcert.csr -CA wpk_root.pem -CAkey wpk_root.key -out wpkcert.pem -CAcreateserial
cd -

Delete unnecessary files

cd ../
rm -rf doc wodles/oscap/content/* gen_ossec.sh add_localfiles.sh Jenkinsfile*
rm -rf src/{addagent,analysisd,client-agent,config,error_messages,external/*,headers,logcollector,monitord,os_auth,os_crypto,os_csyslogd,os_dbdos_execd}
rm -rf src/{os_integrator,os_maild,os_netos_regex,os_xml,os_zlib,remoted,reportd,shared,syscheckd,tests,update,wazuh_db,wazuh_modules}
rm -rf src/win32
rm -rf src/*.a
rm -rf etc/{decoders,lists,rules}
find etc/templates/* -maxdepth 0 -not -name "en" | xargs rm -rf

Compile wpk package

tools/agent-upgrade/wpkpack.py /tmp/myagent.wpk /tmp/wpkcert.pem /tmp/wpkcert.key *

Copy wpk_root.pem to agent

scp -i KEY-PATH /tmp/wpk_root.pem user@ip:/var/ossec/etc

Check that agent is registered

# /var/ossec/bin/agent_control -l

Wazuh agent_control. List of available agents:
   ID: 000, Name: ip-172-31-5-9.ec2.internal (server), IP: 127.0.0.1, Active/Local
   ID: 002, Name: ip-172-31-3-152.ec2.internal, IP: any, Active

Upgrade agent
In the Manager

# /var/ossec/bin/agent_upgrade -a 002 -f /home/qa/myagent.wpk
Upgrading...

Failed upgrades:
	Agent 002 status: Timeout reached while waiting for the response from the agent

In the agent
13. Check upgrade status in log

# tail -f /var/ossec/logs/ossec.log
2022/10/19 14:15:12 wazuh-modulesd:agent-upgrade[3266] wm_agent_upgrade_agent.c:169 at wm_agent_upgrade_listen_messages(): DEBUG: (8156): Response message: '{"error":0,"message":"ok","data":[]}'
2022/10/19 14:15:12 wazuh-agentd[3220] request.c:334 at req_receiver(): DEBUG: req_receiver(): sending '#!-req d405299f {"error":0,"message":"ok","data":[]}' to server
2022/10/19 14:15:12 wazuh-agentd[3220] receiver.c:92 at receive_msg(): DEBUG: Received message: '#!-req d40529a0 upgrade {"command":"write","parameters":{"buffer":"894Yjve1JvIL/62vtT4xS5m9rzS80F8Vf35BeYjLugzaVd/FPrrsG4VzriVchX2HVZ5GF/oOJyjEFfReLwyXpSrSr2TIV+t1PjqLiMs8JQHD9ar1yCaluM6OxPzV/Ba4vb+rVxkfg3cFdA7Zph+kovfWtjiuqruuBzwD5+MbA6ieQ3HtcFxVElprVWkiX9cgXySfrr53pE5RFs+R+CHzTfQrYDwayNFTxCXlmzjhctHeR0xVhktax3FuTyL8r7WttV5pDOTxjZ4L5z8uR/5XRmutdxp9/ythzFwnvS+RRlXvYV3m53Od/MKSafT9wsReOC5jVmutWrYfMzvPpVIPEzsAX/e+uB9906jaVftH5kCIKz8N5X9ZW2tLyXCRuF3th2xLVGYnpHUkXOrB+LpVClprS8vk6ITLRbuaOKtodjUG8QXkWFZmV51wuZrHN+vV+FINxdcNqxSp1aTR74OUNALHVSDyFcCA3ut6fAfnGGxCee76SC0nw0XidrkP+G5lcpTWIfUrOh3qF+ArSKZfTrhclWPntS7hkviyniD0639R2koMnNsJzfG8xzLdrHwDWf0jmW/uBntvrzM0tnTG9Q6o74gaVCx1hjF/KaszlPhx5AN8iuaVT0B978b/zkcwUKdj+oyoo2h5kn8XyHEn3Tp8TVvIn20d8gtbneSbMlA=","length":512,"file":"myagent.wpk"}}'
.
.
.
2022/10/19 14:18:46 wazuh-agentd[3220] request.c:281 at req_receiver(): DEBUG: req_receiver(): sending '{"command":"upgrade","parameters":{"file":"myagent.wpk","installer":"upgrade.sh"}}' to socket
2022/10/19 14:18:46 wazuh-modulesd:agent-upgrade[3266] wm_agent_upgrade_agent.c:165 at wm_agent_upgrade_listen_messages(): DEBUG: (8155): Incomming message: '{"command":"upgrade","parameters":{"file":"myagent.wpk","installer":"upgrade.sh"}}'
.
.
.
2022/10/19 14:20:23 wazuh-modulesd:syscollector[3266] wm_syscollector.c:203 at wm_sys_stop(): INFO: Stop received for Syscollector.
2022/10/19 14:20:23 wazuh-modulesd:syscollector[3266] wm_syscollector.c:191 at wm_sys_main(): INFO: Module finished.
2022/10/19 14:20:23 wazuh-logcollector: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning...
2022/10/19 14:20:23 wazuh-syscheckd: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning...
2022/10/19 14:20:23 wazuh-agentd[3220] sig_op.c:49 at HandleSIG(): INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning...
2022/10/19 14:20:23 wazuh-execd: INFO: (1314): Shutdown received. Deleting responses.
2022/10/19 14:20:23 wazuh-execd: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning...

Check agent's version

/var/ossec/bin/wazuh-control info
WAZUH_VERSION="v4.3.9"
WAZUH_REVISION="40322"
WAZUH_TYPE="agent"

In Manager
15. Add task-timeout on manager to allow more time before task fails

  <task-manager>
    <cleanup_time>1h</cleanup_time>
    <task_timeout>1h</task_timeout>
  </task-manager>

Upgrade agent

#  /var/ossec/bin/agent_upgrade -a 001 -f /tmp/myagent.wpk

Upgrading...

Failed upgrades:
	Agent 001 status: Send upgrade command error

Check logs in manager

2022/10/19 18:28:42 wazuh-modulesd:agent-upgrade[676043] wm_agent_upgrade_manager.c:196 at wm_agent_upgrade_listen_messages(): DEBUG: (8156): Response message: '{"error":0,"data":[{"error":0,"message":"Success","agent":1,"task_id":3,"node":"node01","module":"upgrade_module","command":"upgrade_custom","status":"Error","error_msg":"Send upgrade command error","create_time":"2022/10/19 18:05:21","update_time":"2022/10/19 18:28:40"}],"message":"Success"}'
2022/10/19 18:31:00 wazuh-modulesd:agent-upgrade[676043] wm_agent_upgrade_manager.c:149 at wm_agent_upgrade_listen_messages(): DEBUG: (8155): Incomming message: '{"command":"upgrade_update_status","parameters":{"error":2,"message":"Upgrade failed","status":"Failed","agents":[1]}}'
2022/10/19 18:31:00 wazuh-modulesd:agent-upgrade[676043] wm_agent_upgrade_commands.c:187 at wm_agent_upgrade_process_agent_result_command(): INFO: (8164): Received upgrade notification from agent '1'. Error code: '2', message: 'Upgrade failed'
2022/10/19 18:31:00 wazuh-modulesd:agent-upgrade[676043] wm_agent_upgrade_tasks.c:211 at wm_agent_send_task_information_master(): DEBUG: (8157): Sending message to task_manager module: '{"origin":{"name":"node01","module":"upgrade_module"},"command":"upgrade_update_status","parameters":{"agents":[1],"status":"Failed","error_msg":"Upgrade procedure exited with error code"}}'
2022/10/19 18:31:00 wazuh-modulesd:task-manager[676043] wm_task_manager.c:58 at wm_task_manager_dispatch(): DEBUG: (8204): Incomming message: '{"origin":{"name":"node01","module":"upgrade_module"},"command":"upgrade_update_status","parameters":{"agents":[1],"status":"Failed","error_msg":"Upgrade procedure exited with error code"}}'
2022/10/19 18:31:00 wazuh-modulesd:task-manager[676043] wm_task_manager.c:100 at wm_task_manager_dispatch(): DEBUG: (8205): Response to message: '{"error":0,"data":[{"error":3,"message":"No task in DB","agent":1}],"message":"Success"}'
2022/10/19 18:31:00 wazuh-modulesd:agent-upgrade[676043] wm_agent_upgrade_tasks.c:229 at wm_agent_send_task_information_master(): DEBUG: (8158): Receiving message from task_manager module: '{"error":0,"data":[{"error":3,"message":"No task in DB","agent":1}],"message":"Success"}'
2022/10/19 18:31:00 wazuh-modulesd:agent-upgrade[676043] wm_agent_upgrade_validate.c:453 at wm_agent_upgrade_validate_task_status_message(): ERROR: (8119): There has been an error updating task state. Error code: '3', message: 'No task in DB'
2022/10/19 18:31:00 wazuh-modulesd:agent-upgrade[676043] wm_agent_upgrade_upgrades.c:612 at wm_agent_upgrade_send_command_to_agent(): DEBUG: (8165): Sending message to agent: '001 upgrade {"command":"clear_upgrade_result","parameters":{}}'
2022/10/19 18:31:00 wazuh-modulesd:agent-upgrade[676043] wm_agent_upgrade_upgrades.c:625 at wm_agent_upgrade_send_command_to_agent(): DEBUG: (8166): Receiving message from agent: '{"error":0,"message":"ok","data":[]}'
2022/10/19 18:31:00 wazuh-modulesd:agent-upgrade[676043] wm_agent_upgrade_tasks_callbacks.c:152 at wm_agent_upgrade_update_status_success_callback(): DEBUG: (8167): Upgrade result file has been successfully erased from the agent.
2022/10/19 18:31:00 wazuh-modulesd:agent-upgrade[676043] wm_agent_upgrade_manager.c:196 at wm_agent_upgrade_listen_messages(): DEBUG: (8156): Response message: '{"error":0,"data":[{"error":3,"message":"No task in DB","agent":1}],"message":"Success"}'

In Agent
18. Check upgrade.log results

2022/10/19 18:29:16 - Generating Backup.
tar: ./var/ossec/queue/alerts/execq: socket ignored
tar: ./var/ossec/queue/alerts/cfgaq: socket ignored
tar: ./var/ossec/queue/sockets/logcollector: socket ignored
tar: ./var/ossec/queue/sockets/wmodules: socket ignored
tar: ./var/ossec/queue/sockets/upgrade: socket ignored
tar: ./var/ossec/queue/sockets/control: socket ignored
tar: ./var/ossec/queue/sockets/com: socket ignored
tar: ./var/ossec/queue/sockets/queue: socket ignored
tar: ./var/ossec/queue/sockets/syscheck: socket ignored
2022/10/19 18:29:18 - Upgrade started.

 Wazuh v4.4.0 (Rev. 40400) Installation Script - https://www.wazuh.com

 You are about to start the installation process of Wazuh.
 You must have a C compiler pre-installed in your system.

  - System: Linux ip-172-31-0-169.ec2.internal 5.15.57-30.131.amzn2022.aarch64 (amzn 1.0)
  - User: root
  - Host: ip-172-31-0-169.ec2.internal


  -- Press ENTER to continue or Ctrl-C to abort. --

 - You already have Wazuh installed. Do you want to update it? (y/n): 
    - Installation will be made at  /var/ossec .

4- Installing the system

DIR="/var/ossec"
 - Running the Makefile

Stopping Wazuh...
agent
Wait for success...
success
install: cannot stat 'wazuh_modules/syscollector/norm_config.json': No such file or directory
SCA policies are not available for this OS version amzn 1 0.

Wait for success...
success
Starting Wazuh...
Job for wazuh-agent.service failed because the control process exited with error code.
See "systemctl status wazuh-agent.service" and "journalctl -xeu wazuh-agent.service" for details.

 - Configuration finished properly.

 - To start Wazuh:
      /var/ossec/bin/wazuh-control start

 - To stop Wazuh:
      /var/ossec/bin/wazuh-control stop

 - The configuration can be viewed or modified at /var/ossec/etc/ossec.conf


   Thanks for using Wazuh.
   Please don't hesitate to contact us if you need help or find
   any bugs.

   Use our public Mailing List at:
          https://groups.google.com/forum/#!forum/wazuh

   More information can be found at:
          - http://www.wazuh.com

    ---  Press ENTER to finish (maybe more information below). ---

 - Update completed.

2022/10/19 18:29:39 - Installation result = 0
./var/upgrade/src/init/pkg_installer.sh: line 207: ./var/run/wazuh-agentd.state: No such file or directory
2022/10/19 18:29:40 - Waiting connection... Status = pending. Remaining attempts: 29.
./var/upgrade/src/init/pkg_installer.sh: line 207: ./var/run/wazuh-agentd.state: No such file or directory
2022/10/19 18:29:41 - Waiting connection... Status = pending. Remaining attempts: 28.
./var/upgrade/src/init/pkg_installer.sh: line 207: ./var/run/wazuh-agentd.state: No such file or directory
2022/10/19 18:29:42 - Waiting connection... Status = pending. Remaining attempts: 27.
./var/upgrade/src/init/pkg_installer.sh: line 207: ./var/run/wazuh-agentd.state: No such file or directory
2022/10/19 18:29:43 - Waiting connection... Status = pending. Remaining attempts: 26.
./var/upgrade/src/init/pkg_installer.sh: line 207: ./var/run/wazuh-agentd.state: No such file or directory
2022/10/19 18:29:44 - Waiting connection... Status = pending. Remaining attempts: 25.
./var/upgrade/src/init/pkg_installer.sh: line 207: ./var/run/wazuh-agentd.state: No such file or directory
2022/10/19 18:29:45 - Waiting connection... Status = pending. Remaining attempts: 24.
./var/upgrade/src/init/pkg_installer.sh: line 207: ./var/run/wazuh-agentd.state: No such file or directory
2022/10/19 18:29:46 - Waiting connection... Status = pending. Remaining attempts: 23.
./var/upgrade/src/init/pkg_installer.sh: line 207: ./var/run/wazuh-agentd.state: No such file or directory
2022/10/19 18:29:47 - Waiting connection... Status = pending. Remaining attempts: 22.
./var/upgrade/src/init/pkg_installer.sh: line 207: ./var/run/wazuh-agentd.state: No such file or directory
2022/10/19 18:29:48 - Waiting connection... Status = pending. Remaining attempts: 21.
./var/upgrade/src/init/pkg_installer.sh: line 207: ./var/run/wazuh-agentd.state: No such file or directory
2022/10/19 18:29:49 - Waiting connection... Status = pending. Remaining attempts: 20.
./var/upgrade/src/init/pkg_installer.sh: line 207: ./var/run/wazuh-agentd.state: No such file or directory
2022/10/19 18:29:50 - Waiting connection... Status = pending. Remaining attempts: 19.
./var/upgrade/src/init/pkg_installer.sh: line 207: ./var/run/wazuh-agentd.state: No such file or directory
2022/10/19 18:29:51 - Waiting connection... Status = pending. Remaining attempts: 18.
./var/upgrade/src/init/pkg_installer.sh: line 207: ./var/run/wazuh-agentd.state: No such file or directory
2022/10/19 18:29:52 - Waiting connection... Status = pending. Remaining attempts: 17.
./var/upgrade/src/init/pkg_installer.sh: line 207: ./var/run/wazuh-agentd.state: No such file or directory
2022/10/19 18:29:53 - Waiting connection... Status = pending. Remaining attempts: 16.
./var/upgrade/src/init/pkg_installer.sh: line 207: ./var/run/wazuh-agentd.state: No such file or directory
2022/10/19 18:29:54 - Waiting connection... Status = pending. Remaining attempts: 15.
./var/upgrade/src/init/pkg_installer.sh: line 207: ./var/run/wazuh-agentd.state: No such file or directory
2022/10/19 18:29:55 - Waiting connection... Status = pending. Remaining attempts: 14.
./var/upgrade/src/init/pkg_installer.sh: line 207: ./var/run/wazuh-agentd.state: No such file or directory
2022/10/19 18:29:56 - Waiting connection... Status = pending. Remaining attempts: 13.
./var/upgrade/src/init/pkg_installer.sh: line 207: ./var/run/wazuh-agentd.state: No such file or directory
2022/10/19 18:29:57 - Waiting connection... Status = pending. Remaining attempts: 12.
./var/upgrade/src/init/pkg_installer.sh: line 207: ./var/run/wazuh-agentd.state: No such file or directory
2022/10/19 18:29:58 - Waiting connection... Status = pending. Remaining attempts: 11.
./var/upgrade/src/init/pkg_installer.sh: line 207: ./var/run/wazuh-agentd.state: No such file or directory
2022/10/19 18:29:59 - Waiting connection... Status = pending. Remaining attempts: 10.
./var/upgrade/src/init/pkg_installer.sh: line 207: ./var/run/wazuh-agentd.state: No such file or directory
2022/10/19 18:30:00 - Waiting connection... Status = pending. Remaining attempts: 9.
./var/upgrade/src/init/pkg_installer.sh: line 207: ./var/run/wazuh-agentd.state: No such file or directory
2022/10/19 18:30:01 - Waiting connection... Status = pending. Remaining attempts: 8.
./var/upgrade/src/init/pkg_installer.sh: line 207: ./var/run/wazuh-agentd.state: No such file or directory
2022/10/19 18:30:02 - Waiting connection... Status = pending. Remaining attempts: 7.
./var/upgrade/src/init/pkg_installer.sh: line 207: ./var/run/wazuh-agentd.state: No such file or directory
2022/10/19 18:30:03 - Waiting connection... Status = pending. Remaining attempts: 6.
./var/upgrade/src/init/pkg_installer.sh: line 207: ./var/run/wazuh-agentd.state: No such file or directory
2022/10/19 18:30:04 - Waiting connection... Status = pending. Remaining attempts: 5.
./var/upgrade/src/init/pkg_installer.sh: line 207: ./var/run/wazuh-agentd.state: No such file or directory
2022/10/19 18:30:05 - Waiting connection... Status = pending. Remaining attempts: 4.
./var/upgrade/src/init/pkg_installer.sh: line 207: ./var/run/wazuh-agentd.state: No such file or directory
2022/10/19 18:30:06 - Waiting connection... Status = pending. Remaining attempts: 3.
./var/upgrade/src/init/pkg_installer.sh: line 207: ./var/run/wazuh-agentd.state: No such file or directory
2022/10/19 18:30:07 - Waiting connection... Status = pending. Remaining attempts: 2.
./var/upgrade/src/init/pkg_installer.sh: line 207: ./var/run/wazuh-agentd.state: No such file or directory
2022/10/19 18:30:08 - Waiting connection... Status = pending. Remaining attempts: 1.
./var/upgrade/src/init/pkg_installer.sh: line 207: ./var/run/wazuh-agentd.state: No such file or directory
2022/10/19 18:30:09 - Waiting connection... Status = pending. Remaining attempts: 0.
2022/10/19 18:30:09 - Upgrade failed. Restoring...
wazuh-modulesd not running...
wazuh-logcollector not running...
wazuh-syscheckd not running...
wazuh-agentd not running...
wazuh-execd not running...
Wazuh v4.4.0 Stopped
2022/10/19 18:30:09 - Deleting upgrade files...
2022/10/19 18:30:09 - Restoring backup....
2022/10/19 18:30:09 - Restoring SELinux policy.
2022/10/19 18:30:25 wazuh-modulesd[22819] debug_op.c:70 at _log(): DEBUG: Logging module auto-initialized
2022/10/19 18:30:25 wazuh-modulesd[22819] main.c:76 at main(): DEBUG: Wazuh home directory: /var/ossec
2022/10/19 18:30:25 wazuh-modulesd[22819] wmodules-osquery-monitor.c:78 at wm_osquery_monitor_read(): DEBUG: Logpath read: /var/log/osquery/osqueryd.results.log
2022/10/19 18:30:25 wazuh-modulesd[22819] wmodules-osquery-monitor.c:84 at wm_osquery_monitor_read(): DEBUG: configPath read: /etc/osquery/osquery.conf
2022/10/19 18:30:25 wazuh-modulesd[22819] agent_op.c:218 at os_read_agent_profile(): DEBUG: Calling os_read_agent_profile().
2022/10/19 18:30:25 wazuh-modulesd[22819] agent_op.c:237 at os_read_agent_profile(): DEBUG: os_read_agent_profile() = [amzn, amzn1]
2022/10/19 18:30:25 wazuh-modulesd[22819] config.c:407 at ReadConfig(): DEBUG: agent_config element does not have any attributes.
Starting Wazuh v4.3.8...
Started wazuh-execd...
Started wazuh-agentd...
Started wazuh-syscheckd...
Started wazuh-logcollector...
2022/10/19 18:30:30 wazuh-modulesd[22891] debug_op.c:70 at _log(): DEBUG: Logging module auto-initialized
2022/10/19 18:30:30 wazuh-modulesd[22891] main.c:76 at main(): DEBUG: Wazuh home directory: /var/ossec
2022/10/19 18:30:30 wazuh-modulesd[22891] wmodules-osquery-monitor.c:78 at wm_osquery_monitor_read(): DEBUG: Logpath read: /var/log/osquery/osqueryd.results.log
2022/10/19 18:30:30 wazuh-modulesd[22891] wmodules-osquery-monitor.c:84 at wm_osquery_monitor_read(): DEBUG: configPath read: /etc/osquery/osquery.conf
2022/10/19 18:30:30 wazuh-modulesd[22891] agent_op.c:218 at os_read_agent_profile(): DEBUG: Calling os_read_agent_profile().
2022/10/19 18:30:30 wazuh-modulesd[22891] agent_op.c:237 at os_read_agent_profile(): DEBUG: os_read_agent_profile() = [amzn, amzn1]
2022/10/19 18:30:30 wazuh-modulesd[22891] config.c:407 at ReadConfig(): DEBUG: agent_config element does not have any attributes.
Started wazuh-modulesd...
Completed.

Note: The error present in the Issue is not present but the agent takes too long to update and the manager timeouts, while the agent continues to receive information, it then receives the stop command, to update, but does not update or restart.

Deblintrake09 · 2022-10-18T20:15:02Z

Update 2022/10/17

Setup enviroment and test on aarch64 ALAS2 AMI - could not find ALAS2022 AMI used by core team.
AMI could not connect to local VM or Deployer pipeline VM.

Deblintrake09 · 2022-10-19T12:53:10Z

Update 2022/10/18

Setup new ALAS2 AMI and was able to configure connection
Building WPK package failed while following the steps taken on the Issue.
Got AMI used by core team.
Setup AMI and try to build package. Got Exception when building wpk.
Research cause of error. Fixed error.
Build WPK package.
Tried to upgrade - Failed 🔴

Deblintrake09 · 2022-10-19T12:54:14Z

Update 2022/10/19

Generate new WPK and set up new manager.
Agent device out of space.

2022/10/19 12:11:58 wazuh-modulesd:agent-upgrade[92701] wm_agent_upgrade_parsing.c:486 at wm_agent_upgrade_parse_agent_upgrade_command_response(): ERROR: (8116): Error response from agent: 'File Open Error: No space left on device'
2022/10/19 12:11:58 wazuh-modulesd:agent-upgrade[92701] wm_agent_upgrade_upgrades.c:612 at wm_agent_upgrade_send_command_to_agent(): DEBUG: (8165): Sending message to agent: '001 upgrade {"command":"open","parameters":{"mode":"wb","file":"myagent.wpk"}}'
2022/10/19 12:11:58 wazuh-modulesd:agent-upgrade[92701] wm_agent_upgrade_upgrades.c:625 at wm_agent_upgrade_send_command_to_agent(): DEBUG: (8166): Receiving message from agent: '{"error":6,"message":"File Open Error: No space left on device","data":[]}'
2022/10/19 12:11:58 wazuh-modulesd:agent-upgrade[92701] wm_agent_upgrade_parsing.c:486 at wm_agent_upgrade_parse_agent_upgrade_command_response(): ERROR: (8116): Error response from agent: 'File Open Error: No space left on device'

Setup new environment and try to upgrade agent 🔴 - Timeout Error
Modify config to add task timeout 🔴 - Update fails to restart on the agent and is rolled back

Deblintrake09 · 2022-10-20T17:02:39Z

Testing results

Build WPK and upgrade agent 🔴

Install Manager 4.3.9-1

  Running scriptlet: wazuh-manager-4.3.9-1.x86_64                                1/1 
  Verificando         : wazuh-manager-4.3.9-1.x86_64                                1/1 

Installed:
  wazuh-manager-4.3.9-1.x86_64

In Agent

Install agent 4.3.8

  Running scriptlet: wazuh-agent-4.3.8-1.aarch64                                                                                1/1 
  Verifying        : wazuh-agent-4.3.8-1.aarch64                                                                                1/1 

Installed:
  wazuh-agent-4.3.8-1.aarch64                                                                                                       

Complete!

Install dependencies
Clone Repository

# git clone https://github.com/wazuh/wazuh
# cd wazuh/src
# git checkout fac4fd56911049804c3f1685e6bdcc4e22e926d3

Modify preconfigured vars and uncomment the following lines

# nano etc/preloaded-vars.conf

USER_LANGUAGE="en"
USER_NO_STOP="y"
USER_UPDATE="y"
USER_BINARYINSTALL="y"

Build Agent from sources

    LDFLAGS           '-Wl,-rpath,/../lib' -pthread -lrt -ldl -O2 -Lshared_modules/dbsync/build/lib -Lshared_modules/rsync/build/lib  -Lwazuh_modules/syscollector/build/lib -Ldata_provider/build/lib
    LIBS              -lrt -ldl -lm 
    CC                cc
    MAKE              make
make[1]: se sale del directorio '/home/qa/wazuh/src'

Done building agent

Generate certificates

cd /tmp
openssl req -x509 -new -nodes -newkey rsa:2048 -keyout wpk_root.key -out wpk_root.pem -batch
openssl req -new -nodes -newkey rsa:2048 -keyout wpkcert.key -out wpkcert.csr -subj '/C=US/ST=CA/O=Wazuh'
openssl x509 -req -days 365 -in wpkcert.csr -CA wpk_root.pem -CAkey wpk_root.key -out wpkcert.pem -CAcreateserial
cd -

Delete Unnecesary files

cd ../
rm -rf doc wodles/oscap/content/* gen_ossec.sh add_localfiles.sh Jenkinsfile*
rm -rf src/{addagent,analysisd,client-agent,config,error_messages,external/*,headers,logcollector,monitord,os_auth,os_crypto,os_csyslogd,os_dbdos_execd}
rm -rf src/{os_integrator,os_maild,os_netos_regex,os_xml,os_zlib,remoted,reportd,shared,syscheckd,tests,update,wazuh_db,wazuh_modules}
rm -rf src/win32
rm -rf src/*.a
rm -rf etc/{decoders,lists,rules}
find etc/templates/* -maxdepth 0 -not -name "en" | xargs rm -rf

Build wpk package

tools/agent-upgrade/wpkpack.py /tmp/agent439.wpk /tmp/wpkcert.pem /tmp/wpkcert.key *

Copy wpk file to the manager

scp -i KEY-PATH /tmp/agent439.wpk user@ip:/var/ossec/etc

Check that agent is registered

# /var/ossec/bin/agent_control -l

Wazuh agent_control. List of available agents:
   ID: 000, Name: ip-172-31-5-9.ec2.internal (server), IP: 127.0.0.1, Active/Local
   ID: 001, Name: ip-172-31-0-169.ec2.internal, IP: any, Active

Upgrade the agent from the manager

# /var/ossec/bin/agent_upgrade -a 001 -f /home/qa/agent439.wpk
Agent information not found in database
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/var/ossec/framework/python/lib/python3.9/concurrent/futures/process.py", line 243, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/var/ossec/framework/python/lib/python3.9/site-packages/wazuh-4.3.9-py3.9.egg/wazuh/core/cluster/dapi/dapi.py", line 243, in run_local
    data = f(**f_kwargs)
  File "/var/ossec/framework/python/lib/python3.9/site-packages/wazuh-4.3.9-py3.9.egg/wazuh/rbac/decorators.py", line 420, in wrapper
    result = func(*args, **kwargs) if not skip_execution else None
  File "/var/ossec/framework/python/lib/python3.9/site-packages/wazuh-4.3.9-py3.9.egg/wazuh/agent.py", line 981, in upgrade_agents
    raise WazuhInternalError(error_code, cmd_error=True, extra_message=agent_result['message'])
wazuh.core.exception.WazuhInternalError: Error 1816 - Agent information not found in database
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/ossec/framework/python/lib/python3.9/site-packages/wazuh-4.3.9-py3.9.egg/wazuh/core/cluster/dapi/dapi.py", line 288, in execute_local_request
    data = await asyncio.wait_for(task, timeout=timeout)
  File "/var/ossec/framework/python/lib/python3.9/asyncio/tasks.py", line 442, in wait_for
    return await fut
wazuh.core.exception.WazuhInternalError: Error 1816 - Agent information not found in database
Internal error: 
Traceback (most recent call last):
  File "/var/ossec/framework/scripts/agent_upgrade.py", line 220, in <module>
    main()
  File "/var/ossec/framework/scripts/agent_upgrade.py", line 186, in main
    result = send_command(function=upgrade_agents, command=create_command())
  File "/var/ossec/framework/scripts/agent_upgrade.py", line 115, in send_command
    return raise_if_exc(pool.submit(run, dapi.distribute_function()).result())
  File "/var/ossec/framework/python/lib/python3.9/site-packages/api-4.3.9-py3.9.egg/api/util.py", line 293, in raise_if_exc
    _create_problem(obj)
  File "/var/ossec/framework/python/lib/python3.9/site-packages/api-4.3.9-py3.9.egg/api/util.py", line 263, in _create_problem
    raise ProblemException(status=500 if not code else code, type=exc.type, title=exc.title, detail=exc.message,
connexion.exceptions.ProblemException

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/ossec/framework/scripts/agent_upgrade.py", line 227, in <module>
    if args.debug:
AttributeError: 'Namespace' object has no attribute 'debug'

NOTE: Testing was done in 4.3.8 and 4.3.9 since the Issue was generated with 4.3.8 in mind but it was found that the issue persisted in 4.3.9.

Work on Fixed Branch

Build WPK and upgrade agent through CLI - Service Active 🟡

In the manager

Install Manager

  Ejecutando scriptlet: wazuh-manager-4.4.0-3415.aarch64.wpk.   1/1 
  Verificando         : wazuh-manager-4.4.0-3415.aarch64.wpk.   1/1 

Instalado:
  wazuh-manager-4.4.0-3415.aarch64.wpk.fix.x86_64

In Agent

Install agent 4.3.9-1

# /var/ossec/bin/wazuh-control info
WAZUH_VERSION="v4.3.9"
WAZUH_REVISION="40322"
WAZUH_TYPE="agent"

Install dependencies
Clone repo and modify the etc/preloaded-vars.conf file uncommenting the following lines:

# git clone https://github.com/wazuh/wazuh
# cd wazuh/src
# git checkout 14982-fix-aarch64-not-found
# nano etc/preloaded-vars.conf

USER_LANGUAGE="en"
USER_NO_STOP="y"
USER_UPDATE="y"
USER_BINARYINSTALL="y"

Build agent from sources

# make deps TARGET=agent && make TARGET=agent
    -DOSSECHIDS -DUSER="wazuh" -DGROUPGLOBAL="wazuh" -DLinux -DINOTIFY_ENABLED -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DENABLE_SYSC -DENABLE_CISCAT -DENABLE_AUDIT -DCLIENT
Compiler:
    CFLAGS            -pthread -Iexternal/libdb/build_unix/ -Iexternal/pacman/lib/libalpm/ -Iexternal/libarchive/libarchive -Wl,--start-group -Iexternal/audit-userspace/lib -DNDEBUG -O2 -DOSSECHIDS -DUSER="wazuh" -DGROUPGLOBAL="wazuh" -DLinux -DINOTIFY_ENABLED -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DENABLE_SYSC -DENABLE_CISCAT -DENABLE_AUDIT -DCLIENT -pipe -Wall -Wextra -std=gnu99 -I./ -I./headers/ -Iexternal/openssl/include -Iexternal/cJSON/ -Iexternal/libyaml/include -Iexternal/curl/include -Iexternal/msgpack/include -Iexternal/bzip2/ -Ishared_modules/common -Ishared_modules/dbsync/include -Ishared_modules/rsync/include -Iwazuh_modules/syscollector/include  -Idata_provider/include  -Iexternal/libpcre2/include -Iexternal/rpm//builddir/output/include 
    LDFLAGS           '-Wl,-rpath,/../lib' -pthread -lrt -ldl -O2 -Lshared_modules/dbsync/build/lib -Lshared_modules/rsync/build/lib  -Lwazuh_modules/syscollector/build/lib -Ldata_provider/build/lib
    LIBS              -lrt -ldl -lm 
    CC                cc
    MAKE              make
make[1]: Leaving directory '/home/ec2-user/wazuh/src'

Done building agent

Generate certificate WPK to sign wpk and replace root certificate

cd /tmp
openssl req -x509 -new -nodes -newkey rsa:2048 -keyout wpk_root.key -out wpk_root.pem -batch
openssl req -new -nodes -newkey rsa:2048 -keyout wpkcert.key -out wpkcert.csr -subj '/C=US/ST=CA/O=Wazuh'
openssl x509 -req -days 365 -in wpkcert.csr -CA wpk_root.pem -CAkey wpk_root.key -out wpkcert.pem -CAcreateserial
cd -

Delete unnecessary files

cd ../
rm -rf doc wodles/oscap/content/* gen_ossec.sh add_localfiles.sh Jenkinsfile*
rm -rf src/{addagent,analysisd,client-agent,config,error_messages,external/*,headers,logcollector,monitord,os_auth,os_crypto,os_csyslogd,os_dbdos_execd}
rm -rf src/{os_integrator,os_maild,os_netos_regex,os_xml,os_zlib,remoted,reportd,shared,syscheckd,tests,update,wazuh_db,wazuh_modules}
rm -rf src/win32
rm -rf src/*.a
rm -rf etc/{decoders,lists,rules}
find etc/templates/* -maxdepth 0 -not -name "en" | xargs rm -rf

Compile wpk package

tools/agent-upgrade/wpkpack.py /tmp/fixedagent.wpk /tmp/wpkcert.pem /tmp/wpkcert.key *

Copy wpk_root.pem to agent

scp -i KEY-PATH /tmp/wpk_root.pem user@ip:/var/ossec/etc

Check agent status in systemctl

● wazuh-agent.service - Wazuh agent
     Loaded: loaded (/etc/systemd/system/wazuh-agent.service; enabled; vendor preset: disabled)
     Active: active (running) since Thu 2022-10-20 16:32:25 UTC; 1min 5s ago
    Process: 40604 ExecStart=/usr/bin/env /var/ossec/bin/wazuh-control start (code=exited, status=0/S>
      Tasks: 21 (limit: 2109)
     Memory: 61.4M
        CPU: 7.376s
     CGroup: /system.slice/wazuh-agent.service
             ├─ 40632 /var/ossec/bin/wazuh-execd
             ├─ 40644 /var/ossec/bin/wazuh-agentd
             ├─ 40659 /var/ossec/bin/wazuh-syscheckd
             └─ 40672 /var/ossec/bin/wazuh-logcollector

Check that agent is registered

/var/ossec/bin/agent_control -i 003

Wazuh agent_control. Agent information:
   Agent ID:   003
   Agent Name: ip-172-31-12-133.ec2.internal
   IP address: any
   Status:     Active

   Operating system:    Linux |ip-172-31-12-133.ec2.internal |5.15.57-30.131.amzn2022.aarch64 |#1 SMP Wed Aug 17 23:50:18 UTC 2022 |aarch64
   Client version:      Wazuh v4.3.9
   Configuration hash:  ab73af41699f13fdd81903b5f23d8d00
   Shared file hash:    4a8724b20dee0124ff9656783c490c4e
   Last keep alive:     1666283465

   Syscheck last started at:  Thu Oct 20 16:06:51 2022
   Syscheck last ended at:    Thu Oct 20 16:06:54 2022

Upgrade agent

In the Manager

# /var/ossec/bin/agent_upgrade -a 002 -f /home/qa/fixedagent.wpk
Upgrading...

# cat /var/ossec/log/ossec.log
2022/10/20 16:36:12 wazuh-modulesd:agent-upgrade[137107] wm_agent_upgrade_manager.c:196 at wm_agent_upgrade_listen_messages(): DEBUG: (8156): Response message: '{"error":0,"data":[{"error":0,"message":"Success","agent":3,"task_id":5,"node":"node01","module":"upgrade_module","command":"upgrade_custom","status":"Updating","create_time":"2022/10/20 16:31:23","update_time":"2022/10/20 16:31:23"}],"message":"Success"}'

Check agent's info on manager

# /var/ossec/bin/agent_control -i 003

Wazuh agent_control. Agent information:
   Agent ID:   003
   Agent Name: ip-172-31-12-133.ec2.internal
   IP address: any
   Status:     Active

   Operating system:    Linux |ip-172-31-12-133.ec2.internal |5.15.57-30.131.amzn2022.aarch64 |#1 SMP Wed Aug 17 23:50:18 UTC 2022 |aarch64
   Client version:      Wazuh v4.4.0
   Configuration hash:  ab73af41699f13fdd81903b5f23d8d00
   Shared file hash:    4a8724b20dee0124ff9656783c490c4e
   Last keep alive:     1666283772

   Syscheck last started at:  Thu Oct 20 16:32:20 2022
   Syscheck last ended at:    Thu Oct 20 16:32:23 2022

In Agent

Check upgrade.log results

# cat /var/ossec/logs/upgrade.log 
2022/10/20 16:32:02 - Generating Backup.
tar: ./var/ossec/queue/alerts/cfgaq: socket ignored
tar: ./var/ossec/queue/alerts/execq: socket ignored
tar: ./var/ossec/queue/sockets/com: socket ignored
tar: ./var/ossec/queue/sockets/logcollector: socket ignored
tar: ./var/ossec/queue/sockets/wmodules: socket ignored
tar: ./var/ossec/queue/sockets/control: socket ignored
tar: ./var/ossec/queue/sockets/upgrade: socket ignored
tar: ./var/ossec/queue/sockets/queue: socket ignored
tar: ./var/ossec/queue/sockets/syscheck: socket ignored
2022/10/20 16:32:04 - Upgrade started.

 Wazuh v4.4.0 (Rev. 40400) Installation Script - https://www.wazuh.com

 You are about to start the installation process of Wazuh.
 You must have a C compiler pre-installed in your system.

  - System: Linux ip-172-31-12-133.ec2.internal 5.15.57-30.131.amzn2022.aarch64 (amzn 1.0)
  - User: root
  - Host: ip-172-31-12-133.ec2.internal


  -- Press ENTER to continue or Ctrl-C to abort. --

 - You already have Wazuh installed. Do you want to update it? (y/n): 
    - Installation will be made at  /var/ossec .

4- Installing the system

DIR="/var/ossec"
 - Running the Makefile

Stopping Wazuh...
agent
Wait for success...
success
install: cannot stat 'wazuh_modules/syscollector/norm_config.json': No such file or directory
libsemanage.semanage_get_lock: Could not get direct transaction lock at /var/lib/selinux/targeted/semanage.trans.LOCK. (Resource temporarily unavailable).
semodule:  Failed on /tmp/wazuh.pp!
libsemanage.semanage_get_lock: Could not get direct transaction lock at /var/lib/selinux/targeted/semanage.trans.LOCK. (Resource temporarily unavailable).
semodule:  Failed on wazuh!
SCA policies are not available for this OS version amzn 1 0.


Wait for success...
success
Starting Wazuh...

 - Configuration finished properly.

 - To start Wazuh:
      /var/ossec/bin/wazuh-control start

 - To stop Wazuh:
      /var/ossec/bin/wazuh-control stop

 - The configuration can be viewed or modified at /var/ossec/etc/ossec.conf


   Thanks for using Wazuh.
   Please don't hesitate to contact us if you need help or find
   any bugs.

   Use our public Mailing List at:
          https://groups.google.com/forum/#!forum/wazuh

   More information can be found at:
          - http://www.wazuh.com

    ---  Press ENTER to finish (maybe more information below). ---

 - Update completed.

2022/10/20 16:32:25 - Installation result = 0
2022/10/20 16:32:26 - Waiting connection... Status = connected. Remaining attempts: 29.
2022/10/20 16:32:26 - Connected to manager.
2022/10/20 16:32:26 - Upgrade finished successfully.

NOTE: The CLI upgrade_agent does not finish. Error caused in step 7, some folders needed were removed

Build WPK and upgrade agent through CLI - Service Inactive 🟡

In Agent

Install agent 4.3.9-1

# /var/ossec/bin/wazuh-control info
WAZUH_VERSION="v4.3.9"
WAZUH_REVISION="40322"
WAZUH_TYPE="agent"

Check agent status in systemctl

# systemctl status wazuh-agent.service 
○ wazuh-agent.service - Wazuh agent
     Loaded: loaded (/usr/lib/systemd/system/wazuh-agent.service; disabled; vendor preset: disabled)
     Active: inactive (dead)

Check that agent is registered

/var/ossec/bin/agent_control -i 004

Wazuh agent_control. Agent information:
   Agent ID:   004
   Agent Name: ALAS22-3
   IP address: any
   Status:     Active

   Operating system:    Linux |ip-172-31-12-133.ec2.internal |5.15.57-30.131.amzn2022.aarch64 |#1 SMP Wed Aug 17 23:50:18 UTC 2022 |aarch64
   Client version:      Wazuh v4.3.9
   Configuration hash:  ab73af41699f13fdd81903b5f23d8d00
   Shared file hash:    4a8724b20dee0124ff9656783c490c4e
   Last keep alive:     1666284126

   Syscheck last started at:  Thu Oct 20 16:41:37 2022
   Syscheck last ended at:    Thu Oct 20 16:41:41 2022

Upgrade agent

In the Manager

# /var/ossec/bin/agent_upgrade -a 004 -f /home/qa/fixedagent.wpk
Upgrading...

# cat /var/ossec/log/ossec.log
2022/10/20 16:36:12 wazuh-modulesd:agent-upgrade[137107] wm_agent_upgrade_manager.c:196 at wm_agent_upgrade_listen_messages(): DEBUG: (8156): Response message: '{"error":0,"data":[{"error":0,"message":"Success","agent":4,"task_id":5,"node":"node01","module":"upgrade_module","command":"upgrade_custom","status":"Updating","create_time":"2022/10/20 16:31:23","update_time":"2022/10/20 16:31:23"}],"message":"Success"}'

In the agent

Check upgrade status in log

# cat /var/ossec/log/upgrade.log
2022/10/20 16:43:16 - Generating Backup.
tar: ./var/ossec/queue/alerts/cfgaq: socket ignored
tar: ./var/ossec/queue/alerts/execq: socket ignored
tar: ./var/ossec/queue/sockets/wmodules: socket ignored
tar: ./var/ossec/queue/sockets/logcollector: socket ignored
tar: ./var/ossec/queue/sockets/control: socket ignored
tar: ./var/ossec/queue/sockets/com: socket ignored
tar: ./var/ossec/queue/sockets/upgrade: socket ignored
tar: ./var/ossec/queue/sockets/queue: socket ignored
tar: ./var/ossec/queue/sockets/syscheck: socket ignored
2022/10/20 16:43:18 - Upgrade started.

 Wazuh v4.4.0 (Rev. 40400) Installation Script - https://www.wazuh.com

 You are about to start the installation process of Wazuh.
 You must have a C compiler pre-installed in your system.

  - System: Linux ip-172-31-12-133.ec2.internal 5.15.57-30.131.amzn2022.aarch64 (amzn 1.0)
  - User: root
  - Host: ip-172-31-12-133.ec2.internal


  -- Press ENTER to continue or Ctrl-C to abort. --

 - You already have Wazuh installed. Do you want to update it? (y/n): 
    - Installation will be made at  /var/ossec .

4- Installing the system

DIR="/var/ossec"
 - Running the Makefile

Stopping Wazuh...
agent
Wait for success...
success
install: cannot stat 'wazuh_modules/syscollector/norm_config.json': No such file or directory
libsemanage.semanage_get_lock: Could not get direct transaction lock at /var/lib/selinux/targeted/semanage.trans.LOCK. (Resource temporarily unavailable).
semodule:  Failed on /tmp/wazuh.pp!
libsemanage.semanage_get_lock: Could not get direct transaction lock at /var/lib/selinux/targeted/semanage.trans.LOCK. (Resource temporarily unavailable).
semodule:  Failed on wazuh!
SCA policies are not available for this OS version amzn 1 0.


Wait for success...
success
Starting Wazuh...

 - Configuration finished properly.

 - To start Wazuh:
      /var/ossec/bin/wazuh-control start

 - To stop Wazuh:
      /var/ossec/bin/wazuh-control stop

 - The configuration can be viewed or modified at /var/ossec/etc/ossec.conf


   Thanks for using Wazuh.
   Please don't hesitate to contact us if you need help or find
   any bugs.

   Use our public Mailing List at:
          https://groups.google.com/forum/#!forum/wazuh

   More information can be found at:
          - http://www.wazuh.com

    ---  Press ENTER to finish (maybe more information below). ---

 - Update completed.

2022/10/20 16:43:38 - Installation result = 0
2022/10/20 16:43:39 - Waiting connection... Status = connected. Remaining attempts: 29.
2022/10/20 16:43:39 - Connected to manager.
2022/10/20 16:43:39 - Upgrade finished successfully.

Check agent's info on manager

# /var/ossec/bin/agent_control -i 004

Wazuh agent_control. Agent information:
   Agent ID:   004
   Agent Name: ALAS22-3
   IP address: any
   Status:     Active

   Operating system:    Linux |ip-172-31-12-133.ec2.internal |5.15.57-30.131.amzn2022.aarch64 |#1 SMP Wed Aug 17 23:50:18 UTC 2022 |aarch64
   Client version:      Wazuh v4.4.0
   Configuration hash:  ab73af41699f13fdd81903b5f23d8d00
   Shared file hash:    4a8724b20dee0124ff9656783c490c4e
   Last keep alive:     1666284376

   Syscheck last started at:  Thu Oct 20 16:43:33 2022
   Syscheck last ended at:    Thu Oct 20 16:43:37 2022

Check agentd service status

# systemctl status wazuh-agent.service 
● wazuh-agent.service - Wazuh agent
     Loaded: loaded (/etc/systemd/system/wazuh-agent.service; disabled; vendor preset: disabled)
     Active: active (running) since Thu 2022-10-20 16:43:38 UTC; 3min 44s ago
    Process: 43825 ExecStart=/usr/bin/env /var/ossec/bin/wazuh-control start (code=exited, status=0/S>
      Tasks: 21 (limit: 2109)
     Memory: 8.0M
        CPU: 7.341s
     CGroup: /system.slice/wazuh-agent.service
             ├─ 43853 /var/ossec/bin/wazuh-execd
             ├─ 43865 /var/ossec/bin/wazuh-agentd
             ├─ 43880 /var/ossec/bin/wazuh-syscheckd
             └─ 43893 /var/ossec/bin/wazuh-logcollector

Oct 20 16:43:31 ip-172-31-12-133.ec2.internal systemd[1]: Starting wazuh-agent.service - Wazuh agent.>
Oct 20 16:43:31 ip-172-31-12-133.ec2.internal env[43825]: Starting Wazuh v4.4.0...

NOTE: The CLI upgrade_agent does not finish. Error caused in step 7, some folders needed were removed
NOTE: This test case was done because it was considered that the service being inactive was a possible cause of the bug in agent_upgrade. It was tested to demonstrate the results and check if it was related. It also shows, that after upgrading, the agent is enabled properly.

Build WPK and upgrade agent through API 🟢

In Agent

Install agent 4.3.9-1

# /var/ossec/bin/wazuh-control info
WAZUH_VERSION="v4.3.9"
WAZUH_REVISION="40322"
WAZUH_TYPE="agent"

Check agent status in systemctl

# systemctl status wazuh-agent.service 
● wazuh-agent.service - Wazuh agent
     Loaded: loaded (/usr/lib/systemd/system/wazuh-agent.service; enabled; vendor preset: disabled)
     Active: active (exited) since Thu 2022-10-20 15:25:15 UTC; 13s ago
    Process: 31676 ExecStart=/usr/bin/env /var/ossec/bin/wazuh-control start (code=exited, status=0>
        CPU: 111ms

Oct 20 15:25:12 ip-172-31-12-133.ec2.internal systemd[1]: Starting wazuh-agent.service - Wazuh agen>
Oct 20 15:25:12 ip-172-31-12-133.ec2.internal env[31676]: Starting Wazuh v4.3.9...

Check that agent is registered

/var/ossec/bin/agent_control -i 001

Upgrade agent

In the Manager

# curl -k -X PUT "https://localhost:55000/agents/upgrade_custom?pretty=true&agents_list=all&file_path=/var/ossec/fixedagent.wpk" -H  "Authorization: Bearer $TOKEN"
{
   "data": {
      "affected_items": [
         {
            "agent": "001",
            "task_id": 1
         }
      ],
      "total_affected_items": 1,
      "total_failed_items": 0,
      "failed_items": []
   },
   "message": "All upgrade tasks were created",
   "error": 0
}

In the agent

Check upgrade status in log

# cat /var/ossec/log/upgrade.log
2022/10/20 15:45:41 - Upgrade started.

 Wazuh v4.4.0 (Rev. 40400) Installation Script - https://www.wazuh.com

 You are about to start the installation process of Wazuh.
 You must have a C compiler pre-installed in your system.

  - System: Linux ip-172-31-12-133.ec2.internal 5.15.57-30.131.amzn2022.aarch64 (amzn 1.0)
  - User: root
  - Host: ip-172-31-12-133.ec2.internal


  -- Press ENTER to continue or Ctrl-C to abort. --

 - You already have Wazuh installed. Do you want to update it? (y/n): 
    - Installation will be made at  /var/ossec .

4- Installing the system

DIR="/var/ossec"
 - Running the Makefile

Stopping Wazuh...
agent
Wait for success...
success
install: cannot stat 'wazuh_modules/syscollector/norm_config.json': No such file or directory
libsemanage.semanage_get_lock: Could not get direct transaction lock at /var/lib/selinux/targeted/semanage.trans.LOCK. (Resource temporarily unavailable).
semodule:  Failed on /tmp/wazuh.pp!
libsemanage.semanage_get_lock: Could not get direct transaction lock at /var/lib/selinux/targeted/semanage.trans.LOCK. (Resource temporarily unavailable).
semodule:  Failed on wazuh!
SCA policies are not available for this OS version amzn 1 0.

Wait for success...
success
Starting Wazuh...

 - Configuration finished properly.

 - To start Wazuh:
      /var/ossec/bin/wazuh-control start

 - To stop Wazuh:
      /var/ossec/bin/wazuh-control stop

 - The configuration can be viewed or modified at /var/ossec/etc/ossec.conf

   Thanks for using Wazuh.
   Please don't hesitate to contact us if you need help or find
   any bugs.

   Use our public Mailing List at:
          https://groups.google.com/forum/#!forum/wazuh

   More information can be found at:
          - http://www.wazuh.com

    ---  Press ENTER to finish (maybe more information below). ---

 - Update completed.

2022/10/20 15:46:02 - Installation result = 0
2022/10/20 15:46:03 - Waiting connection... Status = connected. Remaining attempts: 29.
2022/10/20 15:46:03 - Connected to manager.
2022/10/20 15:46:03 - Upgrade finished successfully.

Check agent's version

# /var/ossec/bin/wazuh-control info
WAZUH_VERSION="v4.4.0"
WAZUH_REVISION="40400"
WAZUH_TYPE="agent"

Check agent status in manager

# /var/ossec/bin/agent_control -i 001

Wazuh agent_control. Agent information:
   Agent ID:   001
   Agent Name: ALAS22
   IP address: any
   Status:     Active

   Operating system:    Linux |ip-172-31-12-133.ec2.internal |5.15.57-30.131.amzn2022.aarch64 |#1 SMP Wed Aug 17 23:50:18 UTC 2022 |aarch64
   Client version:      Wazuh v4.4.0
   Configuration hash:  ab73af41699f13fdd81903b5f23d8d00
   Shared file hash:    4a8724b20dee0124ff9656783c490c4e
   Last keep alive:     1666281050

jmv74211 · 2022-10-27T13:49:57Z

Closing conclusion 👍🏼


🔵	Proposed to be fixed in future versions or developments

After talking with the team, the testing has been approved taking into account the following considerations proposed in the QA review:

(1) The steps indicated in the documentation for WPK generation are not correct. 🔵

This is a known bug and it will be fixed in this PR wazuh/wazuh-documentation#5701

Deblintrake09 added team/qa dev-testing labels Oct 4, 2022

Deblintrake09 mentioned this issue Oct 4, 2022

QA Planning: Fix error while upgrading an agent in Amazon Linux 2022 #3404

Closed

4 tasks

Deblintrake09 self-assigned this Oct 4, 2022

Deblintrake09 added feature/wpk role/qa-watchdogs labels Oct 4, 2022

Deblintrake09 changed the title ~~Fix WPK Upgrading on ALAS 2022~~ Fix WPK Update on ALAS 2022 Oct 5, 2022

damarisg changed the title ~~Fix WPK Update on ALAS 2022~~ WPK Update on ALAS 2022 Oct 5, 2022

jmv74211 removed the status/not-tracked label Oct 7, 2022

jmv74211 added this to the QA testing - 4.4.0 milestone Oct 7, 2022

jmv74211 added the target/4.4.0 label Oct 7, 2022

jmv74211 closed this as completed Oct 27, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WPK Update on ALAS 2022 #3415

WPK Update on ALAS 2022 #3415

Deblintrake09 commented Oct 4, 2022 •

edited

Loading

chemamartinez commented Oct 13, 2022

Deblintrake09 commented Oct 17, 2022 •

edited

Loading

Deblintrake09 commented Oct 17, 2022 •

edited

Loading

Deblintrake09 commented Oct 18, 2022 •

edited

Loading

Deblintrake09 commented Oct 19, 2022

Deblintrake09 commented Oct 19, 2022 •

edited

Loading

Deblintrake09 commented Oct 20, 2022 •

edited

Loading

jmv74211 commented Oct 27, 2022 •

edited

Loading

WPK Update on ALAS 2022 #3415

WPK Update on ALAS 2022 #3415

Comments

Deblintrake09 commented Oct 4, 2022 • edited Loading

Description

Proposed checks

Steps to reproduce

Expected results

chemamartinez commented Oct 13, 2022

Deblintrake09 commented Oct 17, 2022 • edited Loading

Review data

Testing environment

Tested packages

Conclusion 🟡

Status

Deblintrake09 commented Oct 17, 2022 • edited Loading

Testing results

Work on unfixed Branch 4.3.9-1

Work on Fixed Branch

Deblintrake09 commented Oct 18, 2022 • edited Loading

Update 2022/10/17

Deblintrake09 commented Oct 19, 2022

Update 2022/10/18

Deblintrake09 commented Oct 19, 2022 • edited Loading

Update 2022/10/19

Deblintrake09 commented Oct 20, 2022 • edited Loading

Testing results

Work on Fixed Branch

jmv74211 commented Oct 27, 2022 • edited Loading

Closing conclusion 👍🏼

Deblintrake09 commented Oct 4, 2022 •

edited

Loading

Deblintrake09 commented Oct 17, 2022 •

edited

Loading

Deblintrake09 commented Oct 17, 2022 •

edited

Loading

Deblintrake09 commented Oct 18, 2022 •

edited

Loading

Deblintrake09 commented Oct 19, 2022 •

edited

Loading

Deblintrake09 commented Oct 20, 2022 •

edited

Loading

jmv74211 commented Oct 27, 2022 •

edited

Loading