[Fleet]: Host Details are not available under predefined filters in the Search bar of Agents Logs tab.

[ghost]

Kibana version: 8.6 Snapshot Kibana cloud environment

Host OS and Browser version: All, All

Build details:

VERSION: 8.6 Snapshot
BUILD: 58830
COMMIT: 6a5d6d96a534be75fc58acda8f89f2610309d7ff
Artifacts: https://artifacts-api.elastic.co/v1/search/8.6-SNAPSHOT

Preconditions:

1. 8.6 BC1 Kibana cloud environment should be available.
2. Agent should be installed.

Steps to reproduce:

1. Login to Kibana environment.
2. Navigate to Fleet tab.
3. Navigate to Logs tab.
4. Search for agent with predefined filter. say' host.hostname
5. Observe host details are not available under filter.

Expected Result:

• Host Details are not available under predefined filters in the Search bar of Agents Logs tab.

What is working fine on 8.6:
Only host.name is working fine.

What is not working fine on 8.6:
i.e. host.id
host.mac,
host.hostname

Screenshots:

What is working fine:
Predefined filters are available under logs tab in 8.5.3
i.e. host.id
host.mac,
host.hostname
host.name

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[dikshachauhan-qasource]

Secondary review is done

[michalpristas]

problem is here
in 8.5:

in 8.6

host metadata have different format and fields, looking into more details

[michalpristas]

looks like add_host_metadata from default config is not taken into account

[AndersonQ]

Here are my findings:

output name is empty, what makes filebeat not load the pipelines and log:

Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. [...]

On my branch I changed the message to include the current configured/enabled output and reduced the level to info. I don' think on a normal situation it'd be a surprise to the user the pipelines being disabled. Anyway, at least I'd say the message must include the current configured/enabled as without it it'd hard to understand what is happening. If we had it this issues would have been a lot more straight forward to debug.

The culprit seems to be BeatConfig which uses output (singular) instead of outputs (plural).

I fixed it. Using a debugger I can see the issue and the fix working, well almost. I does fix the empty output when the configs are loaded on libbeat/cmd/instance/beat.go by func (b *Beat) configure(settings Settings) error.
However when running under agent, it is, or at least at some point it becomes, empty.

How I have debugged it:

• collect a diagnostics
• get fielbeat config: components/filestream-monitoring/filestream-monitoring-agent/beat-rendered-config
• run filebeat with the following parameters:
  • --path.config=/path/to/config/folder
  • -c=beat-rendered-config
  • --path.logs=/some/path
• use dlv exec

 dlv \
--check-go-version=false \
--listen=:4242 \
--headless=true \
--api-version=2 \
--accept-multiclient \
exec /opt/Elastic/Agent/data/elastic-agent-3bff6f/components/filebeat \
-- --path.config=/root/beats/components/filestream-monitoring/filestream-monitoring-agent/ \
-c=beat-rendered-config \
--path.logs=/root/beats/

the double -- -- is correct, the first is for delve know everything after is are parameters for the binary. More on how to debug the agent/beats with delve here and here

So far I can only imagine the issue is the config having changed and libbeat trying to use an old format.

It could be that during the beats initialisation under agent it reads an empty config from disk then the agent send the new one, and I've only fixed when loading from disk. This would explain to work in standalone mode ran through delve.

What I'd try next it to add logs during the config reload process. I'd bet it's the same case as for the initialisation, the code is trying to load the output using a wrong key.

Last thoughts:

• this logs should help to show what has been loaded as output config
• the problem seems to be when parsing the output config like here
• for some reason I was not seeing any log before this line
• having a manual counter here helped me to be sure I was running the right binary. Why? See Fix make clean (#34134) #34141

[cmacknz]

looks like add_host_metadata from default config is not taken into account

Yes the Beats don't actually read the default configuration at startup anymore see #31901. They are likely not creating an add_host_metadata processor instance anymore, nor any of the other default processors in any of the Beat default configurations.

This will be true for each Beat running under agent. The default processors for each Beat are:

• Filebeat:

  beats/x-pack/filebeat/filebeat.yml

  Lines 167 to 173 in d8204a2

  	processors:
  	- add_host_metadata:
  	when.not.contains.tags: forwarded
  	- add_cloud_metadata: ~
  	- add_docker_metadata: ~
  	- add_kubernetes_metadata: ~

• Metricbeat:

  beats/x-pack/metricbeat/metricbeat.yml

  Lines 123 to 127 in d8204a2

  	processors:
  	- add_host_metadata: ~
  	- add_cloud_metadata: ~
  	- add_docker_metadata: ~
  	- add_kubernetes_metadata: ~

• Osquerybeat:

  beats/x-pack/osquerybeat/osquerybeat.yml

  Lines 111 to 113 in d8204a2

  	processors:
  	- add_host_metadata: ~
  	- add_cloud_metadata: ~

• Auditbeat:

  beats/auditbeat/auditbeat.yml

  Lines 151 to 155 in d8204a2

  	processors:
  	- add_host_metadata: ~
  	- add_cloud_metadata: ~
  	- add_docker_metadata: ~

• Packetbeat:

  beats/packetbeat/packetbeat.yml

  Lines 241 to 256 in d8204a2

  	processors:
  	- # Add forwarded to tags when processing data from a network tap or mirror.
  	if.contains.tags: forwarded
  	then:
  	- drop_fields:
  	fields: [host]
  	else:
  	- add_host_metadata: ~
  	- add_cloud_metadata: ~
  	- add_docker_metadata: ~
  	- detect_mime_type:
  	field: http.request.body.content
  	target: http.request.mime_type
  	- detect_mime_type:
  	field: http.response.body.content
  	target: http.response.mime_type

• Heartbeat:

  beats/heartbeat/heartbeat.yml

  Lines 130 to 137 in d8204a2

  	processors:
  	- add_observer_metadata:
  	# Optional, but recommended geo settings for the location Heartbeat is running in
  	#geo:
  	# Token describing this location
  	#name: us-east-1a
  	# Lat, Lon "
  	#location: "37.926868, -78.024902"

The simplest fix for this for the issue reported in just this bug is to add an instance of the add_host_metdata processor to the monitoring filestream configuration. I suspect we would actually want all four of the _metadata processors added by default there.

To restore the default processors used for every Beat now that they aren't read from a configuration file, we could add them in code when we generate the Beat configuration from the UnitExpectedConfig received from the agent as a global processor:

beats/x-pack/libbeat/management/generate.go

Line 155 in acd70e9

func injectGlobalProcesssors(expected *proto.UnitExpectedConfig, stream map[string]interface{}) map[string]interface{} {

However since this default processor set actually varies per Beat we would probably want to do this on a Beat specific basis. If you search the Beats code for SetTransform you will find the place where the Beat specific configuration transformations are added.

For Filebeat it is

beats/x-pack/filebeat/cmd/root.go

Line 22 in d8204a2

management.ConfigTransform.SetTransform(filebeatCfg)

which calls

beats/x-pack/filebeat/cmd/agent.go

Lines 16 to 21 in d8204a2

	func filebeatCfg(rawIn *proto.UnitExpectedConfig, agentInfo *client.AgentInfo) ([]*reload.ConfigWithMeta, error) {
	modules, err := management.CreateInputsFromStreams(rawIn, "logs", agentInfo)
	if err != nil {
	return nil, fmt.Errorf("error creating input list from raw expected config: %w", err)
	}

Doing that for every Beat and testing it is not straight forward.

[ghost]

Hi Team,

We have revalidated this issue on latest 8.6 BC10 kibana cloud environment and found this issue is fixed now.

• Host Details are available under predefined filters in the Search bar of Agents Logs tab.

Build Details:

Version: 8.6.0 BC10
BUILD: 58852
COMMIT: d3a625ef4a6e611a5b3233a1ce5cbe8ef429eb47
Artifacts: https://staging.elastic.co/8.6.0-b6c773f9/summary-8.6.0.html

Screenshots:

Hence, marking this issue as QA:Validated
Thanks