From b3a25057755cd3e00c4b70cbfb1d8f2d460b10fb Mon Sep 17 00:00:00 2001 From: Thiago Padilha Date: Mon, 29 May 2023 16:01:41 -0300 Subject: [PATCH 1/2] input: kafka: Document `data_format` option Signed-off-by: Thiago Padilha --- pipeline/inputs/kafka.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/pipeline/inputs/kafka.md b/pipeline/inputs/kafka.md index aac51620f..b7097ede3 100644 --- a/pipeline/inputs/kafka.md +++ b/pipeline/inputs/kafka.md @@ -9,6 +9,7 @@ This plugin uses the official [librdkafka C library](https://github.com/edenhill | :--- | :--- | :--- | | brokers | Single or multiple list of Kafka Brokers, e.g: 192.168.1.3:9092, 192.168.1.4:9092. | | | topics | Single entry or list of topics separated by comma \(,\) that Fluent Bit will subscribe to. | | +| data\_format | Serialization format of the messages. If set to "json", the payload will be parsed as json. | none | | client\_id | Client id passed to librdkafka. | | | group\_id | Group id passed to librdkafka. | fluent-bit | | poll\_ms | Kafka brokers polling interval in milliseconds. | 500 | @@ -51,6 +52,7 @@ The fluent-bit source repository contains a full example of using fluent-bit to brokers kafka-broker:9092 topics fb-source poll_ms 100 + data_format json [FILTER] Name lua @@ -66,6 +68,8 @@ The fluent-bit source repository contains a full example of using fluent-bit to The above will connect to the broker listening on `kafka-broker:9092` and subscribe to the `fb-source` topic, polling for new messages every 100 milliseconds. +Since the payload will be in json format, we ask the plugin to automatically parse the payload with `data_format json`. + Every message received is then processed with `kafka.lua` and sent back to the `fb-sink` topic of the same broker. The example can be executed locally with `make start` in the `examples/kafka_filter` directory (docker/compose is used). From cfa3006958319f0338b82ac8ab42d0ec54032d04 Mon Sep 17 00:00:00 2001 From: Thiago Padilha Date: Thu, 20 Jul 2023 08:20:43 -0300 Subject: [PATCH 2/2] Rename data_format to format Signed-off-by: Thiago Padilha --- pipeline/inputs/kafka.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/pipeline/inputs/kafka.md b/pipeline/inputs/kafka.md index b7097ede3..68e48d53c 100644 --- a/pipeline/inputs/kafka.md +++ b/pipeline/inputs/kafka.md @@ -9,7 +9,7 @@ This plugin uses the official [librdkafka C library](https://github.com/edenhill | :--- | :--- | :--- | | brokers | Single or multiple list of Kafka Brokers, e.g: 192.168.1.3:9092, 192.168.1.4:9092. | | | topics | Single entry or list of topics separated by comma \(,\) that Fluent Bit will subscribe to. | | -| data\_format | Serialization format of the messages. If set to "json", the payload will be parsed as json. | none | +| format | Serialization format of the messages. If set to "json", the payload will be parsed as json. | none | | client\_id | Client id passed to librdkafka. | | | group\_id | Group id passed to librdkafka. | fluent-bit | | poll\_ms | Kafka brokers polling interval in milliseconds. | 500 | @@ -52,7 +52,7 @@ The fluent-bit source repository contains a full example of using fluent-bit to brokers kafka-broker:9092 topics fb-source poll_ms 100 - data_format json + format json [FILTER] Name lua @@ -68,7 +68,7 @@ The fluent-bit source repository contains a full example of using fluent-bit to The above will connect to the broker listening on `kafka-broker:9092` and subscribe to the `fb-source` topic, polling for new messages every 100 milliseconds. -Since the payload will be in json format, we ask the plugin to automatically parse the payload with `data_format json`. +Since the payload will be in json format, we ask the plugin to automatically parse the payload with `format json`. Every message received is then processed with `kafka.lua` and sent back to the `fb-sink` topic of the same broker.