Rate limit "Cannot index event" log messages

[cmacknz]

beats/libbeat/outputs/elasticsearch/client.go

Lines 487 to 491 in 032a4cf

	// Fatal error and no dead letter index, drop.
	client.log.Warnf("Cannot index event (status=%v): dropping event! Look at the event log to view the event and cause.", itemStatus)
	client.log.Warnw(fmt.Sprintf("Cannot index event %#v (status=%v): %s, dropping event!", event, itemStatus, itemMessage), logp.TypeKey, logp.EventType)
	stats.nonIndexable++
	return false

The "Cannot index event" logs messages are a useful signal in the logs that events are being dropped and (as of 8.15.0) you should look at the local event log for the reason.

Since this log message does not contain any useful debugging information, and has the potential to be generated for every event that flows through the pipeline, there is no value in logging it for each event.

Instead we should rate limit it so that it only appears once in a fixed interval when events are being dropped. The rate limit is initially proposed to be one message every 10 seconds.

The rate limited message should include the number of events that dropped in the current interval. The message can be changed to something like "Failed to index N events in last M seconds. Look at the event log to view the events and cause."

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[jlind23]

@pierrehilbert bumping the priority on this one as it recently had an impact on some users.
cc @lucabelluccini @nimarezainia

[AndersonQ]

@cmacknz, one question, should the report summarise how many events per status code? E.g.:

• 10 events dropped:
  • 5: 402
  • 5: 403

[cmacknz]

It's useful if it is easy to do, if it adds significant complexity I wouldn't bother. The status code will be in the event logs.

[nimarezainia]

after speaking with @pierrehilbert, we would love to celebrate the benefits of the outcomes from this issue. Are we able to quantify the reduction in events/logs sent?

[AndersonQ]

after speaking with @pierrehilbert, we would love to celebrate the benefits of the outcomes from this issue. Are we able to quantify the reduction in events/logs sent?

yes, if we have access to old logs, we can quantify it. I was actually thinking about quantifying it as well and add to the PR, but I got busy with other tasks.

let me try to make a quick and rough estimation

[konnextv]

I came across this error today in a Metricbeat log and am wondering where to find the mentioned event log?
From my understanding I am looking at it and this is where I see the warning as well: /var/log/metricbeat/metricbeat-20241002-7.ndjson

[cmacknz]

You need to be on 8.15+, by default they are next to the regular logs. If you are below 8.15, you can see the cause by turning on debug logging in the regular log files.

See

beats/metricbeat/metricbeat.reference.yml

Line 2458 in e345f28

#=============================== Events Logging ===============================