No debug level logs are generated on changing log level to debug.

[amolnater-qasource]

Kibana version: 8.6 Snapshot kibana cloud-staging environment

Host OS and Browser version: All, All

Build details:

8.6 Snapshot
BUILD: 58830
COMMIT: 6a5d6d96a534be75fc58acda8f89f2610309d7ff
Artifact: https://snapshots.elastic.co/8.6.0-f6d7d537/downloads/beats/elastic-agent/elastic-agent-8.6.0-SNAPSHOT-windows-x86_64.zip

Preconditions:

1. 8.6 Snapshot kibana cloud-staging environment should be available.
2. Agents should be installed.

Steps to reproduce:

1. Navigate to Fleet tab.
2. Update logging level to debug for agents under agent logs.

Logs:
elastic-agent-diagnostics-2022-12-26T07-53-38Z-00.zip

Screenshot:

Related issue:
#1912

Expected Result:
Debug level logs should be generated on changing log level to debug.

[amolnater-qasource]

@manishgupta-qasource Please review.

[manishgupta-qasource]

Secondary review for this ticket is Done

[michalpristas]

cannot reproduce locally.
logs are present in a log file
logs are present in discover, debug filtering works
playing with log level and watching Log Live stream it reflects the settings.
It has a delay though

[amolnater-qasource]

Hi @michalpristas
We have revalidated this issue on latest 8.6 BC9 kibana cloud environment and had below observations:

Observations:

• Agents got unhealthy on changing logging level, issue under Agents goes to Unhealthy state temporarily on changing logging level under Agent Logs tab. #1912
• For windows and mac no debug level logs generated for elastic-agent dataset.
• However debug level logs are available for elastic-agent.filebeat/metricbeat dataset.
• No debug level logs are available for Linux (.tar) agent.

Build details:
BUILD: 58832
COMMIT: 93183bddac40f8a7ee8e566d1651f9f3b586a520

Screen Recording:

Agents.-.Fleet.-.Elastic.-.Google.Chrome.2022-12-29.13-04-28.mp4

Agents.-.Fleet.-.Elastic.-.Google.Chrome.2022-12-29.12-53-27.mp4

Linux Logs:
[Ubuntu]elastic-agent-diagnostics-2022-12-29T07-46-58Z-00.zip

Windows Logs:
elastic-agent-diagnostics-2022-12-29T07-40-02Z-00.zip

Please let us know if anything else is required from our end.

Thanks

[michalpristas]

@amolnater-qasource could you revalidate with SNAPSHOT?

[amolnater-qasource]

Hi @michalpristas
We have observed this issue as still reproducible on current available 8.6 SNAPSHOT.

Observations:

• Debug logs are not available on changing log level.
• On manually restarting agent service/VM, debug level agent logs are observed.
• Detailed testing observations with agent logs are shared under comment at Agents goes to Unhealthy state temporarily on changing logging level under Agent Logs tab. #1912 (comment)

Please let us know if anything else is required from our end.
Thanks

[cmacknz]

The agent here is affected by elastic/beats#34137 which will be fixed in BC10.

[cmacknz]

The filestream monitoring instance is being left without an output configuration because of the bug in elastic/beats#34137

    state: 2
    message: 'Healthy: communicating with pid ''2435'''
    units:
      ? unittype: 0
        unitid: filestream-monitoring-filestream-monitoring-agent
      : state: 4
        message: '[output unit has no config]'
      ? unittype: 1
        unitid: filestream-monitoring
      : state: 4
        message: '[output unit has no config]'

The filestream monitoring component is restarted before transitioning to the failed state because the initial configuration is provided with no configuration.

{"log.level":"info","@timestamp":"2022-12-29T07:27:46.819Z","message":"beat is restarting because output changed","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log.logger":"centralmgmt.V2-manager","log.origin":{"file.line":503,"file.name":"management/managerV2.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-12-29T07:27:57.411Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":833},"message":"Unit state changed filestream-monitoring-filestream-monitoring-agent (STARTING->FAILED): [output unit has no config]","component":{"id":"filestream-monitoring","state":"HEALTHY"},"unit":{"id":"filestream-monitoring-filestream-monitoring-agent","type":"input","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-12-29T07:27:57.411Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":833},"message":"Unit state changed filestream-monitoring (STARTING->FAILED): [output unit has no config]","component":{"id":"filestream-monitoring","state":"HEALTHY"},"unit":{"id":"filestream-monitoring","type":"output","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}

[amolnater-qasource]

Hi @cmacknz
We have revalidated this issue on latest 8.6 BC10 Kibana cloud environment and found it partially fixed.

Observations:

• Agents remain Healthy on changing log level to debug, validated under Agents goes to Unhealthy state temporarily on changing logging level under Agent Logs tab. #1912.
• Debug Logs are generated for linux agents.
• Debug level logs are generated for Windows and MAC agents for elastic_agent.filebeat/metricbeat datasets.
• On manually restarting Windows and MAC agents debug logs are generated for elastic_agent dataset.

Build details:
BUILD: 58852
COMMIT: d3a625ef4a6e611a5b3233a1ce5cbe8ef429eb47
Artifacts: https://staging.elastic.co/8.6.0-b6c773f9/summary-8.6.0.html#elastic-agent

Screen Recording:

Agents.-.Fleet.-.Elastic.-.Google.Chrome.2023-01-05.15-14-05.mp4

After reboot:

Before reboot Logs:
elastic-agent-diagnostics-2023-01-05T09-47-54Z-00.zip

Please let us know if anything else is required from our end.
Thanks

[cmacknz]

Thanks the missing agent debug logs are likely caused by #1955, the agent also has few messages it logs at the debug level as part of normal operation.

[amolnater-qasource]

Thank you for confirming @cmacknz
Hence, we are marking this as QA:Validated.