Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Provide an indication that a line is a multiline. #957

Closed
incogniro opened this issue Feb 10, 2016 · 2 comments
Closed

Provide an indication that a line is a multiline. #957

incogniro opened this issue Feb 10, 2016 · 2 comments
Assignees
@kvch
Labels
enhancement Filebeat Filebeat

Comments

@incogniro
Copy link

When a multiline is processed and shipped by Filebeat, add to 'tags' an indicator. This is to mimic the behaviour the logstash multiline filter performs by adding 'multiline' to tags.
Originally reported on the discuss forum.
@ruflin
Copy link
Member

ruflin commented Mar 16, 2016 •
edited
Loading

I was having a quick look on this on how we could implement it. It is a little bit more complicated as I thought as the reader is completely decoupled from the event creation. One solution could be to add a Info() method to the LineProcessor interface. This would then return the type of the last line read.

Instead of using a tag I would suggest to add a multiline field which as content has the number of lines which were combined. Like this we directly have some additional meta information.

@alpha-centauri
Copy link

+1 for the idea of adding a multiline field with the number of lines

@ruflin ruflin mentioned this issue Aug 16, 2016
Closed
ruflin added a commit to ruflin/beats that referenced this issue Sep 19, 2016
@ruflin
Add line number counter for multiline
bbd195e 
This allows to indicate if an event was multiline or not. The number of lines will be put under the multiline namespace and looks as following:

```
{
  ...
  "message": "[2015] hello world\n  First Line\n  Second Line",
  "multiline": {
    "lines": 3
  },
  ...
}
```

See elastic#957

= Refactor fields handling with readers

Each reader can add fields to the message object. The reader itself should always add data under its own namespace to prevent conflicts. All these fields are then added to the Data object. This will allow each reader in the future to add its own data if needed.

The JSON reader was simplified in the way that data by default is written under the `json` namespace. Now no special fields have to be passed for JSON and the processing can still happen on the event level.

Further refactoring to the JSON processing should happen in an other PR as event is probably not the right place to happen, as also the JSON config should not be part of it.
ruflin added a commit to ruflin/beats that referenced this issue Sep 25, 2017
@ruflin
Add line number counter for multiline
1865dcf 
This allows to indicate if an event was multiline or not. The number of lines will be put under the multiline namespace and looks as following:

```
{
  ...
  "message": "[2015] hello world\n  First Line\n  Second Line",
  "multiline": {
    "lines": 3
  },
  ...
}
```

See elastic#957
@kvch kvch mentioned this issue Aug 17, 2018
Merged
kvch added a commit to kvch/beats that referenced this issue Aug 30, 2018
@kvch
add log.multiline if message consists of multiple lines
6e822b3 
Closes elastic#957
kvch added a commit that referenced this issue Aug 31, 2018
@kvch
Add tag "multiline" to "log.flags" if event consists of multiple line…
6da83e8 
…s. (#7997)

Add "multiline" tag to "log.status" if the event contains multiple lines. This way users can filter for multiline messages using "multiline" in [log.status].

Example event

{
  "@timestamp": "2018-08-17T11:35:21.813Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "7.0.0-alpha1"
  },
  "source": "/home/n/test.log",
  "offset": 0,
  "log": {
    "status": [
        "multiline"
     ],
  },
  "message": "[test line\ntest line]",
  "prospector": {
    "type": "log"
  },
  "input": {
    "type": "log"
  },
  "beat": {
    "hostname": "sleipnir",
    "version": "7.0.0-alpha1",
    "name": "sleipnir"
  },
  "host": {
    "name": "sleipnir"
  }
}

Closes #957
@kvch kvch mentioned this issue Sep 3, 2018
Merged
kvch added a commit to kvch/beats that referenced this issue Sep 11, 2018
@kvch
Add tag "multiline" to "log.flags" if event consists of multiple line…
cb73c73 
…s. (elastic#7997)

Add "multiline" tag to "log.status" if the event contains multiple lines. This way users can filter for multiline messages using "multiline" in [log.status].

Example event

{
  "@timestamp": "2018-08-17T11:35:21.813Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "7.0.0-alpha1"
  },
  "source": "/home/n/test.log",
  "offset": 0,
  "log": {
    "status": [
        "multiline"
     ],
  },
  "message": "[test line\ntest line]",
  "prospector": {
    "type": "log"
  },
  "input": {
    "type": "log"
  },
  "beat": {
    "hostname": "sleipnir",
    "version": "7.0.0-alpha1",
    "name": "sleipnir"
  },
  "host": {
    "name": "sleipnir"
  }
}

Closes elastic#957

(cherry picked from commit 6da83e8)
kvch added a commit that referenced this issue Sep 11, 2018
@kvch
Add tag "multiline" to "log.flags" if event consists of multiple line…
4d3ccd6 
…s. (#7997) (#8207)

Add "multiline" tag to "log.status" if the event contains multiple lines. This way users can filter for multiline messages using "multiline" in [log.status].

Example event

{
  "@timestamp": "2018-08-17T11:35:21.813Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "7.0.0-alpha1"
  },
  "source": "/home/n/test.log",
  "offset": 0,
  "log": {
    "status": [
        "multiline"
     ],
  },
  "message": "[test line\ntest line]",
  "prospector": {
    "type": "log"
  },
  "input": {
    "type": "log"
  },
  "beat": {
    "hostname": "sleipnir",
    "version": "7.0.0-alpha1",
    "name": "sleipnir"
  },
  "host": {
    "name": "sleipnir"
  }
}

Closes #957

(cherry picked from commit 6da83e8)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kvch kvch

Labels
enhancement Filebeat Filebeat
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@ruflin @alpha-centauri @kvch @andrewkroh @incogniro