[Stack 8.16.0-SNAPSHOT] [claroty_ctd] Failing test daily: system test: tcp in claroty_ctd.event

[elastic-vault-github-plugin-prod]

• Stack version: 8.16.0-SNAPSHOT
• Package: claroty_ctd
• Failing test: system test: tcp
• DataStream: event
• Owners:
  • @elastic/security-service-integrations

Failure:

test case failed: the test service claroty-ctd-tcp unexpectedly exited with code 1

First build failed: https://buildkite.com/elastic/integrations/builds/15613

Latest 5 failed builds:

• https://buildkite.com/elastic/integrations/builds/15629
• https://buildkite.com/elastic/integrations/builds/15682
• https://buildkite.com/elastic/integrations/builds/15733
• https://buildkite.com/elastic/integrations/builds/15809
• https://buildkite.com/elastic/integrations/builds/15864

[efd6]

This failure again looks like a general CI infrastructure failure (ref #10620). The only failure that can be seen from the logs is in the ability to reach a network.

claroty-ctd-tcp-1  | {"level":"info","ts":"2024-09-07T01:50:47.589Z","caller":"command/root.go:147","msg":"Waiting for signal.","start-signal":"SIGHUP"}
claroty-ctd-tcp-1  | {"level":"info","ts":"2024-09-07T01:50:57.932Z","caller":"command/root.go:158","msg":"Delaying connection.","delay":5}
claroty-ctd-tcp-1  | {"level":"debug","ts":"2024-09-07T01:51:02.933Z","caller":"output/util.go:28","msg":"Connecting...","address":"elastic-agent:9537"}
claroty-ctd-tcp-1  | {"level":"debug","ts":"2024-09-07T01:51:03.936Z","caller":"output/util.go:28","msg":"Connecting...","address":"elastic-agent:9537"}
claroty-ctd-tcp-1  | {"level":"debug","ts":"2024-09-07T01:51:04.936Z","caller":"output/util.go:28","msg":"Connecting...","address":"elastic-agent:9537"}
claroty-ctd-tcp-1  | {"level":"debug","ts":"2024-09-07T01:51:05.937Z","caller":"output/util.go:28","msg":"Connecting...","address":"elastic-agent:9537"}
claroty-ctd-tcp-1  | {"level":"debug","ts":"2024-09-07T01:51:06.938Z","caller":"output/util.go:28","msg":"Connecting...","address":"elastic-agent:9537"}
claroty-ctd-tcp-1  | {"level":"debug","ts":"2024-09-07T01:51:07.940Z","caller":"output/util.go:28","msg":"Connecting...","address":"elastic-agent:9537"}
claroty-ctd-tcp-1  | {"level":"debug","ts":"2024-09-07T01:51:08.942Z","caller":"output/util.go:28","msg":"Connecting...","address":"elastic-agent:9537"}
claroty-ctd-tcp-1  | {"level":"debug","ts":"2024-09-07T01:51:09.943Z","caller":"output/util.go:28","msg":"Connecting...","address":"elastic-agent:9537"}
claroty-ctd-tcp-1  | {"level":"debug","ts":"2024-09-07T01:51:10.945Z","caller":"output/util.go:28","msg":"Connecting...","address":"elastic-agent:9537"}
claroty-ctd-tcp-1  | {"level":"debug","ts":"2024-09-07T01:51:11.946Z","caller":"output/util.go:28","msg":"Connecting...","address":"elastic-agent:9537"}
claroty-ctd-tcp-1  | Error: dial tcp 172.23.0.2:9537: connect: connection refused

[efd6]

@mrodm Was able to repro this (thanks), so after further investigation, I can see that the TCP input fails to start with

{
    "log.level": "error",
    "@timestamp": "2024-09-11T10:40:32.431Z",
    "message": "Input 'tcp' failed with: failed to unpack the replace configuration: string value is not set accessing 'processors.6.replace.fields.0.replacement'",
    "component": {
        "binary": "filebeat",
        "dataset": "elastic_agent.filebeat",
        "id": "tcp-default",
        "type": "tcp"
    },
    "log": {
        "source": "tcp-default"
    },
    "log.logger": "input.tcp",
    "log.origin": {
        "file.line": 139,
        "file.name": "compat/compat.go",
        "function": "github.com/elastic/beats/v7/filebeat/input/v2/compat.(*runner).Start.func1"
    },
    "service.name": "filebeat",
    "id": "tcp-claroty_ctd.event-8b22ab77-f96e-439e-944a-da15285c3289",
    "ecs.version": "1.6.0"
}

This is referring to the replace processor.

integrations/packages/claroty_ctd/data_stream/event/agent/stream/tcp.yml.hbs

Lines 19 to 27 in 50fbe53

	- replace:
	fields:
	- field: "message"
	pattern: '\\'
	- field: "message"
	pattern: "Alert/"
	replacement: "Alert-"
	ignore_missing: false
	fail_on_error: true

This is rendered into the following beat config (inputs only).

inputs:
    - data_stream:
        dataset: claroty_ctd.event
        type: logs
      host: 0.0.0.0:9537
      id: tcp-claroty_ctd.event-8b22ab77-f96e-439e-944a-da15285c3289
      index: logs-claroty_ctd.event-91014
      processors:
        - add_fields:
            fields:
                input_id: tcp-claroty_ctd-8b22ab77-f96e-439e-944a-da15285c3289
            target: '@metadata'
        - add_fields:
            fields:
                dataset: claroty_ctd.event
                namespace: "91014"
                type: logs
            target: data_stream
        - add_fields:
            fields:
                dataset: claroty_ctd.event
            target: event
        - add_fields:
            fields:
                stream_id: tcp-claroty_ctd.event-8b22ab77-f96e-439e-944a-da15285c3289
            target: '@metadata'
        - add_fields:
            fields:
                id: ce100c8d-9bc8-43f3-8bd4-3da6f3387042
                snapshot: true
                version: 8.16.0
            target: elastic_agent
        - add_fields:
            fields:
                id: ce100c8d-9bc8-43f3-8bd4-3da6f3387042
            target: agent
        - replace:
            fail_on_error: true
            fields:
                - field: message
                  pattern: \\
                - field: message
                  pattern: Alert/
                  replacement: Alert-
            ignore_missing: false
        - decode_cef:
            ecs: false
            field: message
        - rename:
            fields:
                - from: message
                  to: event.original
      publisher_pipeline:
        disable_host: true
      tags:
        - preserve_original_event
        - preserve_duplicate_custom_fields
        - forwarded
        - claroty_ctd-event
      type: tcp

This is failing due to a missing replacement field. Prior to elastic/beats#40047 this was silently ignored. That PR added a "required" validation to the config.