Filebeat crash with "panic: runtime error: invalid memory address or nil pointer dereference"

[eedugon]

Under specific circumstances and configuration filebeat crashes with the following error:

-- Unit filebeat.service has finished starting up.
--
-- The start-up result is done.
Jul 31 16:31:10 lx-sanboxserver filebeat[15240]: panic: runtime error: invalid memory address or nil pointer dereference
Jul 31 16:31:10 lx-sanboxserver filebeat[15240]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1b74a8d]
Jul 31 16:31:10 lx-sanboxserver filebeat[15240]: goroutine 380 [running]:
Jul 31 16:31:10 lx-sanboxserver filebeat[15240]: github.com/elastic/beats/libbeat/processors/convert.(*processor).writeToEvent(0xc0000fb4f0, 0xc002f9ab80, 0xc002f9ab80, 0x0)
Jul 31 16:31:10 lx-sanboxserver filebeat[15240]: /go/src/github.com/elastic/beats/libbeat/processors/convert/convert.go:148 +0xdd
Jul 31 16:31:10 lx-sanboxserver filebeat[15240]: github.com/elastic/beats/libbeat/processors/convert.(*processor).Run(0xc0000fb4f0, 0xc002f9ab80, 0x0, 0x0, 0x0)
Jul 31 16:31:10 lx-sanboxserver filebeat[15240]: /go/src/github.com/elastic/beats/libbeat/processors/convert/convert.go:102 +0x128
Jul 31 16:31:10 lx-sanboxserver filebeat[15240]: github.com/elastic/beats/libbeat/publisher/processing.(*group).Run(0xc000691620, 0xc002f9ab80, 0xc002f9ab80, 0x0, 0x0)
Jul 31 16:31:10 lx-sanboxserver filebeat[15240]: /go/src/github.com/elastic/beats/libbeat/publisher/processing/processors.go:104 +0x94
Jul 31 16:31:10 lx-sanboxserver filebeat[15240]: github.com/elastic/beats/libbeat/publisher/processing.(*group).Run(0xc000955950, 0xc002f9ab80, 0xc000e07ca8, 0x30d1880, 0x65866a1ce8598e01)
Jul 31 16:31:10 lx-sanboxserver filebeat[15240]: /go/src/github.com/elastic/beats/libbeat/publisher/processing/processors.go:104 +0x94
Jul 31 16:31:10 lx-sanboxserver filebeat[15240]: github.com/elastic/beats/libbeat/publisher/pipeline.(*client).publish(0xc00093ad80, 0x0, 0xed6b6141e, 0x0, 0xc0030cae70, 0xc0030cadb0, 0x0, 0x0, 0x0)
Jul 31 16:31:10 lx-sanboxserver filebeat[15240]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/client.go:89 +0x571
Jul 31 16:31:10 lx-sanboxserver filebeat[15240]: github.com/elastic/beats/libbeat/publisher/pipeline.(*client).Publish(0xc00093ad80, 0x0, 0xed6b6141e, 0x0, 0xc0030cae70, 0xc0030cadb0, 0x0, 0x0, 0x0)
Jul 31 16:31:10 lx-sanboxserver filebeat[15240]: /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/client.go:68 +0xc2
Jul 31 16:31:10 lx-sanboxserver filebeat[15240]: github.com/elastic/beats/filebeat/channel.(*outlet).OnEvent(0xc000955cb0, 0x0, 0xed6b6141e, 0x0, 0xc0030cae70, 0xc0030cadb0, 0x0, 0x0, 0x0, 0xc0030cad80)
Jul 31 16:31:10 lx-sanboxserver filebeat[15240]: /go/src/github.com/elastic/beats/filebeat/channel/outlet.go:64 +0x7e
Jul 31 16:31:10 lx-sanboxserver filebeat[15240]: github.com/elastic/beats/filebeat/harvester.(*Forwarder).Send(0xc000951a20, 0x0, 0xed6b6141e, 0x0, 0xc0030cae70, 0xc0030cadb0, 0x0, 0x0, 0x0, 0x0, ...)
Jul 31 16:31:10 lx-sanboxserver filebeat[15240]: /go/src/github.com/elastic/beats/filebeat/harvester/forwarder.go:50 +0x65
Jul 31 16:31:10 lx-sanboxserver filebeat[15240]: github.com/elastic/beats/filebeat/input/syslog.NewInput.func1(0xc003264800, 0x207, 0x2800, 0x371a9c0, 0xc0030cad80, 0x371a900, 0x0)
Jul 31 16:31:10 lx-sanboxserver filebeat[15240]: /go/src/github.com/elastic/beats/filebeat/input/syslog/input.go:132 +0x127
Jul 31 16:31:10 lx-sanboxserver filebeat[15240]: github.com/elastic/beats/filebeat/inputsource/udp.(*Server).run(0xc00089bec0)
Jul 31 16:31:10 lx-sanboxserver filebeat[15240]: /go/src/github.com/elastic/beats/filebeat/inputsource/udp/server.go:133 +0x19d
Jul 31 16:31:10 lx-sanboxserver filebeat[15240]: github.com/elastic/beats/filebeat/inputsource/udp.(*Server).Start.func1(0xc00089bec0)
Jul 31 16:31:10 lx-sanboxserver filebeat[15240]: /go/src/github.com/elastic/beats/filebeat/inputsource/udp/server.go:85 +0x61
Jul 31 16:31:10 lx-sanboxserver filebeat[15240]: created by github.com/elastic/beats/filebeat/inputsource/udp.(*Server).Start
Jul 31 16:31:10 lx-sanboxserver filebeat[15240]: /go/src/github.com/elastic/beats/filebeat/inputsource/udp/server.go:83 +0x170
Jul 31 16:31:10 lx-sanboxserver systemd[1]: filebeat.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Jul 31 16:31:10 lx-sanboxserver systemd[1]: Unit filebeat.service entered failed state.
Jul 31 16:31:10 lx-sanboxserver systemd[1]: filebeat.service failed.
Jul 31 16:31:11 lx-sanboxserver systemd[1]: filebeat.service holdoff time over, scheduling restart.
Jul 31 16:31:11 lx-sanboxserver systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
-- Subject: Unit filebeat.service has finished shutting down

This crash is occurring when:

• A convert processor is configured at global level (not at the input level), like:

processors:
  - convert:
      fields:
        - {from: "Info", to: "Info", type: "string"}
      ignore_missing: true
      fail_on_error: false

• Some inputs are configured which source raw data include that referenced Info field.
• Other modules are configured, for example Panw and System modules. Note the data related with these modules won't have that Info field.
• There's a significant input pressure (i know this is a bit vague and difficult to determine).

The crash is not occurring when:

• The convert processor is moved to the specific input / inputs sections that might require them (which I would say it's the good practice here).
• The problem is also not occurring when no extra modules are configured, just the inputs that are related with the Info field. Then it doesn't matter if convert is configured at global level.

We suspect the problem could be related with a race condition because when Filebeat is not under high input pressure the problem doesn't occur either, only when we provide a lot of data to Filebeat via big input files, then Filebeat crashes.

Issue has been reproduced with RHEL 7.8 and Windows Server 2016 with Filebeat 7.4.0 and 7.6.1.

Still not clear the root cause, apparently it's not related with a broken / invalid JSON.
I suspect more that maybe the convert processor is applied towards a memory structure that doesn't have the Info field, or a memory structure that is totally missing).

I'm reporting this after a conversation with @sayden

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[rev0lt]

Hey we are facing the exact same issue due to the convert processor, filebeats version 7.4.0, happy to share further details if this will be addressed

[jlind23]

@rev0lt did you have a chance to try a most recent version of filebeat like a 7.17.X?

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[rev0lt]

Hi @jlind23 i havent, is there a possible fix in those versions? This is a bit of effort for us but wouldnt mind doing this if it fixes the issue.

[jlind23]

@rev0lt I would always recommend to update, even if there is no fix for this particular behaviour you will benefit for a lot of other bugfixes. I will add it to our backlog in order to investigate to check if this is still occuring in further releases.

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!