Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move parsers outside of filestream input so others can use them as well #26541

Merged
merged 5 commits into from
Jun 29, 2021

Conversation

kvch
Copy link
Contributor

@kvch kvch commented Jun 28, 2021

What does this PR do?

The object has its own Unpack function, so it is enough for you to add it as an attribute to your configuration.

config parser.Config

Then create the parser based on the configuration

p = inp.parserConfig.Create(r)

Example configuration accepted by the code above

parsers:
  - multiline:
      type: count
      lines_count: 3

Why is it important?

This lets all inputs use the generic parsers of libbeat. The available parsers can be changed by passing a list of allowed parsers for the input.

Checklist

  • My code follows the style guidelines of this project
    - [ ] I have commented my code, particularly in hard-to-understand areas
    - [ ] I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
    - [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 28, 2021
@kvch kvch requested a review from urso June 28, 2021 18:02
@kvch kvch added the Team:Elastic-Agent Label for the Agent team label Jun 28, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/agent (Team:Agent)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 28, 2021
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 28, 2021

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #26541 updated

  • Start Time: 2021-06-29T12:53:26.462+0000

  • Duration: 28 min 50 sec

  • Commit: 8394f29

Trends 🧪

Image of Build Times

Test errors 0

Expand to view the tests failures

Build&Test / filebeat-build / TestParsersMultilineMaxLines – github.com/elastic/beats/v7/filebeat/input/filestream
    Expand to view the error details

     Failed 
    

    Expand to view the stacktrace

     === RUN   TestParsersMultilineMaxLines
    --- FAIL: TestParsersMultilineMaxLines (0.11s)
    panic: interface conversion: interface {} is nil, not string [recovered]
    	panic: interface conversion: interface {} is nil, not string
    
    goroutine 476 [running]:
    testing.tRunner.func1.2(0xbf1ec0, 0xc0002d0420)
    	/usr/local/go/src/testing/testing.go:1143 +0x332
    testing.tRunner.func1(0xc0003c0180)
    	/usr/local/go/src/testing/testing.go:1146 +0x4b6
    panic(0xbf1ec0, 0xc0002d0420)
    	/usr/local/go/src/runtime/panic.go:965 +0x1b9
    github.com/elastic/beats/v7/filebeat/input/filestream.(*inputTestingEnvironment).requireEventsReceived(0xc0003c4460, 0xc000611f40, 0x3, 0x3)
    	/go/src/github.com/elastic/beats/filebeat/input/filestream/environment_test.go:320 +0x4f4
    github.com/elastic/beats/v7/filebeat/input/filestream.TestParsersMultilineMaxLines(0xc0003c0180)
    	/go/src/github.com/elastic/beats/filebeat/input/filestream/parsers_integration_test.go:323 +0x765
    testing.tRunner(0xc0003c0180, 0xd19190)
    	/usr/local/go/src/testing/testing.go:1193 +0xef
    created by testing.(*T).Run
    	/usr/local/go/src/testing/testing.go:1238 +0x2b3
     
    

Steps errors 2

Expand to view the steps failures

libbeat-lint - make -C libbeat check;make -C libbeat update;make -C x-pack/libbeat check;make -C
  • Took 1 min 28 sec . View more details on here
  • Description: make -C libbeat check;make -C libbeat update;make -C x-pack/libbeat check;make -C x-pack/libbeat update;make check-no-changes;
Error signal
  • Took 0 min 0 sec . View more details on here
  • Description: Error 'hudson.AbortException: script returned exit code 2'

Log output

Expand to view the last 100 lines of log output

[2021-06-29T13:20:55.213Z] + find . -type d -name vendor -exec rm -r {} ;
[2021-06-29T13:20:55.521Z] + python .ci/scripts/pre_archive_test.py
[2021-06-29T13:20:55.521Z] Copy ./x-pack/packetbeat/build into build/x-pack/packetbeat/build
[2021-06-29T13:20:55.533Z] Running in /var/lib/jenkins/workspace/PR-26541-5-f9fb3b80-97a9-4690-a1d1-aac0c1e51203/src/github.com/elastic/beats/build
[2021-06-29T13:20:55.549Z] Recording test results
[2021-06-29T13:20:55.762Z] None of the test reports contained any result
[2021-06-29T13:20:55.762Z] [Checks API] No suitable checks publisher found.
[2021-06-29T13:20:56.115Z] + go clean -modcache
[2021-06-29T13:20:58.954Z] Cleaning up /var/lib/jenkins/workspace/PR-26541-5-f9fb3b80-97a9-4690-a1d1-aac0c1e51203
[2021-06-29T13:20:58.954Z] Client: Docker Engine - Community
[2021-06-29T13:20:58.954Z]  Version:           20.10.3
[2021-06-29T13:20:58.954Z]  API version:       1.41
[2021-06-29T13:20:58.954Z]  Go version:        go1.13.15
[2021-06-29T13:20:58.954Z]  Git commit:        48d30b5
[2021-06-29T13:20:58.954Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-06-29T13:20:58.954Z]  OS/Arch:           linux/amd64
[2021-06-29T13:20:58.954Z]  Context:           default
[2021-06-29T13:20:58.954Z]  Experimental:      true
[2021-06-29T13:20:58.954Z] 
[2021-06-29T13:20:58.954Z] Server: Docker Engine - Community
[2021-06-29T13:20:58.954Z]  Engine:
[2021-06-29T13:20:58.954Z]   Version:          20.10.3
[2021-06-29T13:20:58.954Z]   API version:      1.41 (minimum version 1.12)
[2021-06-29T13:20:58.954Z]   Go version:       go1.13.15
[2021-06-29T13:20:58.954Z]   Git commit:       46229ca
[2021-06-29T13:20:58.954Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-06-29T13:20:58.954Z]   OS/Arch:          linux/amd64
[2021-06-29T13:20:58.954Z]   Experimental:     false
[2021-06-29T13:20:58.954Z]  containerd:
[2021-06-29T13:20:58.954Z]   Version:          1.4.3
[2021-06-29T13:20:58.954Z]   GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
[2021-06-29T13:20:58.954Z]  runc:
[2021-06-29T13:20:58.954Z]   Version:          1.0.0-rc92
[2021-06-29T13:20:58.954Z]   GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
[2021-06-29T13:20:58.954Z]  docker-init:
[2021-06-29T13:20:58.954Z]   Version:          0.19.0
[2021-06-29T13:20:58.954Z]   GitCommit:        de40ad0
[2021-06-29T13:20:58.954Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-06-29T13:20:59.891Z] Change permissions with write access of all files inside the specific folder
[2021-06-29T13:20:59.915Z] Running in /var/lib/jenkins/workspace/PR-26541-5-f9fb3b80-97a9-4690-a1d1-aac0c1e51203
[2021-06-29T13:21:02.909Z] + gsutil --version
[2021-06-29T13:21:04.329Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-06-29T13:21:04.636Z] + gcloud auth activate-service-account --key-file ****
[2021-06-29T13:21:05.203Z] Activated service account credentials for: [[email protected]]
[2021-06-29T13:21:05.518Z] + gsutil -m -q cp -a public-read eC1wYWNrL3BhY2tldGJlYXQtbGludDgzOTRmMjkwMTMzZGQ5ZThlYWY5ZDU1N2Y3ZjExYWUxYWQ3ZTNhNTY gs://beats-ci-temp/ci/cache/
[2021-06-29T13:21:07.259Z] + go clean -modcache
[2021-06-29T13:21:10.115Z] Cleaning up /var/lib/jenkins/workspace/Beats_beats_PR-26541
[2021-06-29T13:21:10.115Z] Client: Docker Engine - Community
[2021-06-29T13:21:10.115Z]  Version:           20.10.3
[2021-06-29T13:21:10.115Z]  API version:       1.41
[2021-06-29T13:21:10.115Z]  Go version:        go1.13.15
[2021-06-29T13:21:10.115Z]  Git commit:        48d30b5
[2021-06-29T13:21:10.115Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-06-29T13:21:10.115Z]  OS/Arch:           linux/amd64
[2021-06-29T13:21:10.115Z]  Context:           default
[2021-06-29T13:21:10.115Z]  Experimental:      true
[2021-06-29T13:21:10.115Z] 
[2021-06-29T13:21:10.115Z] Server: Docker Engine - Community
[2021-06-29T13:21:10.115Z]  Engine:
[2021-06-29T13:21:10.115Z]   Version:          20.10.3
[2021-06-29T13:21:10.115Z]   API version:      1.41 (minimum version 1.12)
[2021-06-29T13:21:10.115Z]   Go version:       go1.13.15
[2021-06-29T13:21:10.115Z]   Git commit:       46229ca
[2021-06-29T13:21:10.115Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-06-29T13:21:10.115Z]   OS/Arch:          linux/amd64
[2021-06-29T13:21:10.115Z]   Experimental:     false
[2021-06-29T13:21:10.115Z]  containerd:
[2021-06-29T13:21:10.115Z]   Version:          1.4.3
[2021-06-29T13:21:10.115Z]   GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
[2021-06-29T13:21:10.115Z]  runc:
[2021-06-29T13:21:10.115Z]   Version:          1.0.0-rc92
[2021-06-29T13:21:10.115Z]   GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
[2021-06-29T13:21:10.115Z]  docker-init:
[2021-06-29T13:21:10.115Z]   Version:          0.19.0
[2021-06-29T13:21:10.115Z]   GitCommit:        de40ad0
[2021-06-29T13:21:10.115Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-06-29T13:21:10.115Z] Unable to find image 'alpine:3.4' locally
[2021-06-29T13:21:10.763Z] 3.4: Pulling from library/alpine
[2021-06-29T13:21:11.044Z] c1e54eec4b57: Pulling fs layer
[2021-06-29T13:21:11.327Z] c1e54eec4b57: Verifying Checksum
[2021-06-29T13:21:11.327Z] c1e54eec4b57: Download complete
[2021-06-29T13:21:11.327Z] c1e54eec4b57: Pull complete
[2021-06-29T13:21:11.327Z] Digest: sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c
[2021-06-29T13:21:11.327Z] Status: Downloaded newer image for alpine:3.4
[2021-06-29T13:21:12.283Z] Change permissions with write access of all files inside the specific folder
[2021-06-29T13:21:12.585Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-26541
[2021-06-29T13:21:15.005Z] Stage "Build&Test" skipped due to earlier failure(s)
[2021-06-29T13:21:15.037Z] Stage "Extended" skipped due to earlier failure(s)
[2021-06-29T13:21:15.065Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-06-29T13:21:15.096Z] Stage "Packaging-Pipeline" skipped due to earlier failure(s)
[2021-06-29T13:21:15.144Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-26541/src/github.com/elastic/beats
[2021-06-29T13:21:15.444Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-26541
[2021-06-29T13:21:15.481Z] [INFO] getVaultSecret: Getting secrets
[2021-06-29T13:21:15.523Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-06-29T13:21:16.244Z] + chmod 755 generate-build-data.sh
[2021-06-29T13:21:16.244Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26541/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26541/runs/5 FAILURE 1669519
[2021-06-29T13:21:16.244Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26541/runs/5/steps/?limit=10000 -o steps-info.json
[2021-06-29T13:21:18.106Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26541/runs/5/tests/?status=FAILED -o tests-errors.json
[2021-06-29T13:21:18.106Z] Retry 1/3 exited 22, retrying in 1 seconds...
[2021-06-29T13:21:18.656Z] Retry 2/3 exited 22, retrying in 2 seconds...

❕ Flaky test report

No test was executed to be analysed.

libbeat/parser/parser.go Outdated Show resolved Hide resolved
type Config struct {
pCfg CommonConfig
parsers []common.ConfigNamespace
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe consider to implement Unpack on the config. Then it can be used directly with go-ucfg unpack like this (we have a similar pattern for TLS and I just introduced one for http):

type myConfig struct {

  Parser parser.Config `config:",inline"`
}

with

func (c *Config) Unpack(cfg *common.Config) error {
  tmp := struct {
    Common CommonConfig `config:",inline"`
    Parsers []common.ConfigNamespace `config:"parser"`
  }{}

  if err := cfg.Unpack(&tmp); err != nil {
    return err
  }

  newC, err := NewConfig(tmp.Common, tmp.Parsers)
  if err != nil {
    return err
  }

  *c = *newC
  return nil
}

That way if we have an error, the full name of the setting that failed to parse will be pointed out.

Plus users can use it like:

var settings myConfig
if err := cfg.Unpack(&settings); err != nil {
  return err
}

reader := ...
contentsReader := settings.Parser.Create(reader)

@@ -71,7 +72,7 @@ type readerConfig struct {
MaxBytes int `config:"message_max_bytes" validate:"min=0,nonzero"`
Tail bool `config:"seek_to_tail"`

Parsers []common.ConfigNamespace `config:"parsers"`
Parsers parser.Config `config:",inline"`
Copy link

@urso urso Jun 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would clash with MaxBytes and LineTerminator, right?

Would it make sense to move Exclude/Include lines or encoding to the parser as well?

@kvch kvch requested a review from urso June 28, 2021 19:29
@kvch kvch force-pushed the feature-libbeat-unified-parser-init branch from 3befb94 to f01f797 Compare June 29, 2021 12:04
@kvch kvch merged commit 2a56cd7 into elastic:master Jun 29, 2021
@kvch kvch added the backport-v7.14.0 Automated backport with mergify label Jun 29, 2021
mergify bot pushed a commit that referenced this pull request Jun 29, 2021
…ll (#26541)

## What does this PR do?

The object has its own `Unpack` function, so it is enough for you to add it as an attribute to your configuration.

```golang
config parser.Config
```

Then create the parser based on the configuration

```golang
p = inp.parserConfig.Create(r)
```

Example configuration accepted by the code above

```yaml
parsers:
  - multiline:
      type: count
      lines_count: 3
```

(cherry picked from commit 2a56cd7)
@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #26541 updated

  • Start Time: 2021-06-29T13:22:31.824+0000

  • Duration: 146 min 47 sec

  • Commit: eefea28

Test stats 🧪

Test Results
Failed 0
Passed 47857
Skipped 5392
Total 53249

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 47857
Skipped 5392
Total 53249

v1v added a commit to v1v/beats that referenced this pull request Jun 29, 2021
…arwin-arm64

* upstream/master: (295 commits)
  Update urllib to 1.26.5. (elastic#26380)
  Update golang.org/x/crypto (elastic#26448)
  [Filebeat] Update Fortinet Ingest Pipeline (elastic#24816)
  Move parsers outside of filestream input so others can use them as well (elastic#26541)
  [Filebeat] Fix `threatintel.indicator.url.full` field not populating (elastic#26508)
  [Filebeat] Add network direction processor to Zeek and Suricata modules (elastic#24620)
  Logging code cleanup related to Nomad auto-discovery (elastic#26498)
  [Metricbeat] Add Couchbase's Sync Gateway module (elastic#25599)
  Refactor add_cloud_metadata to handle ECS fields easier (elastic#26438)
  [Elastic Agent] Improper casting of int64 (elastic#26520)
  [Elastic Agent] Enable configuring monitoring namespace (elastic#26439)
  [Heartbeat] configure permissions for synthetics config (elastic#26393)
  Osquerybeat: set the raw index name to supress the timestamp suffix (elastic#26545)
  [Heartbeat] add screenshots config to synthetics (elastic#26455)
  [Elastic Agent] Use http2 to connect to Fleet Server. (elastic#26474)
  Remove all docs about  Beats central management (elastic#26399)
  update data.json for gcp billing (elastic#26506)
  Skip x-pack metricbeat tests (elastic#26537)
  [Elastic Agent] Fix issue with FLEET_CA not being used with Fleet Server in container (elastic#26529)
  Add changelog entry for  elastic#26224 (elastic#26531)
  ...
kvch added a commit that referenced this pull request Jun 29, 2021
…ll (#26541) (#26571)

## What does this PR do?

The object has its own `Unpack` function, so it is enough for you to add it as an attribute to your configuration.

```golang
config parser.Config
```

Then create the parser based on the configuration

```golang
p = inp.parserConfig.Create(r)
```

Example configuration accepted by the code above

```yaml
parsers:
  - multiline:
      type: count
      lines_count: 3
```

(cherry picked from commit 2a56cd7)

Co-authored-by: Noémi Ványi <[email protected]>
mdelapenya added a commit to mdelapenya/beats that referenced this pull request Jun 30, 2021
* master: (25 commits)
  fix: Force PLATFORMS environment variable when we build Elastic Agent dependencies on arm64 (elastic#26415)
  macos for metricbeat to run in the extended meta-stage (elastic#26573)
  Packaging: add arm7 platform in the main pipeline (elastic#26575)
  [Heartbeat] Skip flakey timer queue test (elastic#26592)
  Update to "read_pipeline" permission (elastic#26465) (elastic#26580)
  API keys do not reflect the need for read_pipeline (elastic#26466) (elastic#26582)
  Add Fleet agent.id to Agent monitoring data (elastic#26548)
  Add kinesis metricset (elastic#25989)
  Refactor of system/memory metricset (elastic#26334)
  Introduce httpcommon package in libbeat (add support for Proxy) (elastic#25219)
  [Filebeat] change multiline configuration in awss3 input to parsers (elastic#25873)
  docs: Hint for the error "Error extracting container id" (elastic#25824)
  [Docs] Fixed metricbeat redis exported field CPU descriptions (elastic#25846) (elastic#26496)
  Update urllib to 1.26.5. (elastic#26380)
  Update golang.org/x/crypto (elastic#26448)
  [Filebeat] Update Fortinet Ingest Pipeline (elastic#24816)
  Move parsers outside of filestream input so others can use them as well (elastic#26541)
  [Filebeat] Fix `threatintel.indicator.url.full` field not populating (elastic#26508)
  [Filebeat] Add network direction processor to Zeek and Suricata modules (elastic#24620)
  Logging code cleanup related to Nomad auto-discovery (elastic#26498)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-v7.14.0 Automated backport with mergify Team:Elastic-Agent Label for the Agent team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants