Skip to content

Commit

Permalink
Add documentation for new trace command line. (#1166)
Browse files Browse the repository at this point in the history 
Document the new trace command line options introduced by fluent/fluent-bit#6726.

Signed-off-by: Phillip Whelan <[email protected]>
  • Loading branch information
@pwhelan
pwhelan authored Jul 25, 2023
1 parent ae337ad commit 2393c88
Showing 1 changed file with 113 additions and 4 deletions.
117 changes: 113 additions & 4 deletions administration/troubleshooting.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,19 +8,128 @@
Tap can be used to generate events or records detailing what messages
pass through Fluent Bit, at what time and what filters affect them.


### Simple example

First, we will make sure that the container image we are going to use actually supports Fluent Bit Tap (available in Fluent Bit 2.0+):

```shell
$ docker run --rm -ti fluent/fluent-bit:latest --help | grep -Z
-Z, --enable-chunk-trace enable chunk tracing. activating it requires using the HTTP Server API.
$ docker run --rm -ti fluent/fluent-bit:latest --help | grep trace
-Z, --enable-chunk-traceenable chunk tracing, it can be activated either through the http api or the command line
--trace-input input to start tracing on startup.
--trace-output output to use for tracing on startup.
--trace-output-property set a property for output tracing on startup.
--trace setup a trace pipeline on startup. Uses a single line, ie: "input=dummy.0 output=stdout output.format='json'"
```

If the `--enable-chunk-trace` option is present it means Fluent Bit has support for Fluent Bit Tap but it is disabled by default, so remember to enable it with this option.

Tap support is enabled and disabled via the embedded web server, so enable it like so (or the equivalent option in the configuration file):
You can start fluent-bit with tracing activated from the beginning by using the `trace-input` and `trace-output` properties, like so:

```bash
$ fluent-bit -Z -i dummy -o stdout -f 1 --trace-input=dummy.0 --trace-output=stdout
Fluent Bit v2.1.8
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/07/21 16:27:01] [ info] [fluent bit] version=2.1.8, commit=824ba3dd08, pid=622937
[2023/07/21 16:27:01] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/07/21 16:27:01] [ info] [cmetrics] version=0.6.3
[2023/07/21 16:27:01] [ info] [ctraces ] version=0.3.1
[2023/07/21 16:27:01] [ info] [input:dummy:dummy.0] initializing
[2023/07/21 16:27:01] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2023/07/21 16:27:01] [ info] [sp] stream processor started
[2023/07/21 16:27:01] [ info] [output:stdout:stdout.0] worker #0 started
[2023/07/21 16:27:01] [ info] [fluent bit] version=2.1.8, commit=824ba3dd08, pid=622937
[2023/07/21 16:27:01] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/07/21 16:27:01] [ info] [cmetrics] version=0.6.3
[2023/07/21 16:27:01] [ info] [ctraces ] version=0.3.1
[2023/07/21 16:27:01] [ info] [input:emitter:trace-emitter] initializing
[2023/07/21 16:27:01] [ info] [input:emitter:trace-emitter] storage_strategy='memory' (memory only)
[2023/07/21 16:27:01] [ info] [sp] stream processor started
[2023/07/21 16:27:01] [ info] [output:stdout:stdout.0] worker #0 started
.[0] dummy.0: [[1689971222.068537501, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1689971223.068556121, {}], {"message"=>"dummy"}]
[0] trace: [[1689971222.068677045, {}], {"type"=>1, "trace_id"=>"0", "plugin_instance"=>"dummy.0", "records"=>[{"timestamp"=>1689971222, "record"=>{"message"=>"dummy"}}], "start_time"=>1689971222, "end_time"=>1689971222}]
[1] trace: [[1689971222.068735577, {}], {"type"=>3, "trace_id"=>"0", "plugin_instance"=>"dummy.0", "records"=>[{"timestamp"=>1689971222, "record"=>{"message"=>"dummy"}}], "start_time"=>1689971222, "end_time"=>1689971222}]
[0] dummy.0: [[1689971224.068586317, {}], {"message"=>"dummy"}]
[0] trace: [[1689971223.068626923, {}], {"type"=>1, "trace_id"=>"1", "plugin_instance"=>"dummy.0", "records"=>[{"timestamp"=>1689971223, "record"=>{"message"=>"dummy"}}], "start_time"=>1689971223, "end_time"=>1689971223}]
[1] trace: [[1689971223.068675735, {}], {"type"=>3, "trace_id"=>"1", "plugin_instance"=>"dummy.0", "records"=>[{"timestamp"=>1689971223, "record"=>{"message"=>"dummy"}}], "start_time"=>1689971223, "end_time"=>1689971223}]
[2] trace: [[1689971224.068689341, {}], {"type"=>1, "trace_id"=>"2", "plugin_instance"=>"dummy.0", "records"=>[{"timestamp"=>1689971224, "record"=>{"message"=>"dummy"}}], "start_time"=>1689971224, "end_time"=>1689971224}]
[3] trace: [[1689971224.068747182, {}], {"type"=>3, "trace_id"=>"2", "plugin_instance"=>"dummy.0", "records"=>[{"timestamp"=>1689971224, "record"=>{"message"=>"dummy"}}], "start_time"=>1689971224, "end_time"=>1689971224}]
^C[2023/07/21 16:27:05] [engine] caught signal (SIGINT)
[2023/07/21 16:27:05] [ warn] [engine] service will shutdown in max 5 seconds
[2023/07/21 16:27:05] [ info] [input] pausing dummy.0
[0] dummy.0: [[1689971225.068568875, {}], {"message"=>"dummy"}]
[2023/07/21 16:27:06] [ info] [engine] service has stopped (0 pending tasks)
[2023/07/21 16:27:06] [ info] [input] pausing dummy.0
[2023/07/21 16:27:06] [ warn] [engine] service will shutdown in max 1 seconds
[0] trace: [[1689971225.068654038, {}], {"type"=>1, "trace_id"=>"3", "plugin_instance"=>"dummy.0", "records"=>[{"timestamp"=>1689971225, "record"=>{"message"=>"dummy"}}], "start_time"=>1689971225, "end_time"=>1689971225}]
[1] trace: [[1689971225.068695829, {}], {"type"=>3, "trace_id"=>"3", "plugin_instance"=>"dummy.0", "records"=>[{"timestamp"=>1689971225, "record"=>{"message"=>"dummy"}}], "start_time"=>1689971225, "end_time"=>1689971225}]
[2023/07/21 16:27:07] [ info] [engine] service has stopped (0 pending tasks)
[2023/07/21 16:27:07] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/07/21 16:27:07] [ info] [output:stdout:stdout.0] thread worker #0 stopped
[2023/07/21 16:27:07] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/07/21 16:27:07] [ info] [output:stdout:stdout.0] thread worker #0 stopped
```

If you see the following warning then the `-Z` or `--enable-chunk-tracing` option is missing:

```bash
[2023/07/21 16:26:42] [ warn] [chunk trace] enable chunk tracing via the configuration or command line to be able to activate tracing.
```

Properties can be set for the output using the `--trace-output-property` option:

```bash
$ fluent-bit -Z -i dummy -o stdout -f 1 --trace-input=dummy.0 --trace-output=stdout --trace-output-property=format=json_lines
Fluent Bit v2.1.8
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/07/21 16:28:59] [ info] [fluent bit] version=2.1.8, commit=824ba3dd08, pid=623170
[2023/07/21 16:28:59] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/07/21 16:28:59] [ info] [cmetrics] version=0.6.3
[2023/07/21 16:28:59] [ info] [ctraces ] version=0.3.1
[2023/07/21 16:28:59] [ info] [input:dummy:dummy.0] initializing
[2023/07/21 16:28:59] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2023/07/21 16:28:59] [ info] [sp] stream processor started
[2023/07/21 16:28:59] [ info] [output:stdout:stdout.0] worker #0 started
[2023/07/21 16:28:59] [ info] [fluent bit] version=2.1.8, commit=824ba3dd08, pid=623170
[2023/07/21 16:28:59] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/07/21 16:28:59] [ info] [cmetrics] version=0.6.3
[2023/07/21 16:28:59] [ info] [ctraces ] version=0.3.1
[2023/07/21 16:28:59] [ info] [input:emitter:trace-emitter] initializing
[2023/07/21 16:28:59] [ info] [input:emitter:trace-emitter] storage_strategy='memory' (memory only)
[2023/07/21 16:29:00] [ info] [sp] stream processor started
[2023/07/21 16:29:00] [ info] [output:stdout:stdout.0] worker #0 started
.[0] dummy.0: [[1689971340.068565891, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1689971341.068632477, {}], {"message"=>"dummy"}]
{"date":1689971340.068745,"type":1,"trace_id":"0","plugin_instance":"dummy.0","records":[{"timestamp":1689971340,"record":{"message":"dummy"}}],"start_time":1689971340,"end_time":1689971340}
{"date":1689971340.068825,"type":3,"trace_id":"0","plugin_instance":"dummy.0","records":[{"timestamp":1689971340,"record":{"message":"dummy"}}],"start_time":1689971340,"end_time":1689971340}
[0] dummy.0: [[1689971342.068613646, {}], {"message"=>"dummy"}]
```

With that options set the stdout plugin is now emitting traces in `json_lines` format:

```json
{"date":1689971340.068745,"type":1,"trace_id":"0","plugin_instance":"dummy.0","records":[{"timestamp":1689971340,"record":{"message":"dummy"}}],"start_time":1689971340,"end_time":1689971340}
```

All three options can also be defined using the much more flexible `--trace` option:

```bash
$ fluent-bit -Z -i dummy -o stdout -f 1 --trace="input=dummy.0 output=stdout output.format=json_lines"
```

We defined the entire tap pipeline using this configuration: `input=dummy.0 output=stdout output.format=json_lines` which defines the following:

* input: dummy.0 (listens to the tag and/or alias `dummy.0`)
* output: stdout (outputs to a stdout plugin)
* output.format: json_lines (sets the stdout format o `json_lines`)

Tap support can also be activated and deactivated via the embedded web server:

```shell
$ docker run --rm -ti -p 2020:2020 fluent/fluent-bit:latest -Z -H -i dummy -p alias=input_dummy -o stdout -f 1
Expand Down

0 comments on commit 2393c88

Please sign in to comment.