Enable TLS in filebeat #101

VariableDeclared · 2022-11-22T16:51:41Z

Add TLS Support to the filebeat charm

Thanks,
Peter

Added client TLS to filebeat to enable pushing to Graylog. TLS Client is only used to install the PKI CA and does not request certificates of its own.

agileshaw

Overall LGTM. Just one question about the build image change

agileshaw · 2022-11-28T21:53:22Z

charmcraft.yaml

@@ -15,7 +15,7 @@ parts:
 bases:
 - build-on:
 - name: ubuntu
- channel: "20.04"
+ channel: "22.04"


I see that you've changed the build base to 22.04. Have you tested the charm's behavior and confirmed that it is working as intended?

yes, I can change this back - it was a issue with my workstation I was working around :-)

Actually testing again, so it appears that the build is broken on 20.04, but working on 22.04

Could you please attach the output for the 20.04 build?

sure: https://pastebin.ubuntu.com/p/YVbrhyNRmB/

VariableDeclared · 2022-12-05T13:08:32Z

hey @agileshaw any advice on the focal/jammy build issue?

agileshaw · 2022-12-05T21:09:48Z

Hi @VariableDeclared . Apologize for the late reply.

This is a known issue with Jinja and MarkupSafe2 in focal. A workaround can be found here: juju/charm-tools#650

VariableDeclared · 2022-12-06T17:15:21Z

hey @agileshaw had a look, but that fix is not merged - I cannot see a workaround there, what's your suggestion?

Adding the arguement is not yet working:

2022-12-06 17:13:53.433 :: 2022-12-06 17:13:52.940 :: usage: charm-build [-h] [-l LOG_LEVEL] [-f] [-o OUTPUT_DIR] [-d BUILD_DIR]
2022-12-06 17:13:53.433 :: 2022-12-06 17:13:52.940 ::                    [-C CACHE_DIR] [-s SERIES] [--hide-metrics]
2022-12-06 17:13:53.433 :: 2022-12-06 17:13:52.940 ::                    [--interface-service INTERFACE_SERVICE] [-i LAYER_INDEX]
2022-12-06 17:13:53.433 :: 2022-12-06 17:13:52.940 ::                    [--no-local-layers] [-n NAME] [--write-lock-file]
2022-12-06 17:13:53.433 :: 2022-12-06 17:13:52.940 ::                    [--use-lock-file-branches] [--ignore-lock-file] [-r] [-R]
2022-12-06 17:13:53.433 :: 2022-12-06 17:13:52.940 ::                    [-w WHEELHOUSE_OVERRIDES] [-W] [-v] [--debug] [-c]
2022-12-06 17:13:53.433 :: 2022-12-06 17:13:52.940 ::                    [--charm-file] [--binary-wheels]
2022-12-06 17:13:53.433 :: 2022-12-06 17:13:52.940 ::                    [--binary-wheels-from-source] [--use-python-from-snap]
2022-12-06 17:13:53.433 :: 2022-12-06 17:13:52.940 ::                    [--description]
2022-12-06 17:13:53.433 :: 2022-12-06 17:13:52.940 ::                    [charm]
2022-12-06 17:13:53.433 :: 2022-12-06 17:13:52.940 :: charm-build: error: unrecognized arguments: --upgrade-buildvenv-core-deps

agileshaw · 2022-12-06T17:30:13Z

Since the issue only seems to appear on focal, I think we can use jammy as build base (so you don't need to change back the metadata.yaml). Please attach the log of a jammy-built charm and test its behavior with some simple bundle (e.g. https://git.launchpad.net/charm-graylog/tree/src/tests/functional/tests/bundles/base-graylog.yaml). If everything checks out, that's a +1 from my book.

I'm also adding @esunar as a reviewer.

lathiat · 2022-12-09T01:41:36Z

The fix here may work:
https://discourse.charmhub.io/t/install-or-update-python-packages-before-packing-a-charm/5158
canonical/charmcraft#551

parts:
  charm:
    charm-python-packages:
      # https:/canonical/charmcraft/issues/551
      - setuptools

VariableDeclared · 2022-12-09T14:58:42Z

hey @lathiat it appears at some point this arg was removed:

➜  filebeat-charm git:(lma-improvements/graylog-tls) ✗ charmcraft version
Bad charmcraft.yaml content:
- extra field 'charm-python-packages' not permitted in 'parts.charm' configuration
Full execution log: '/home/pjds/snap/charmcraft/common/cache/charmcraft/log/charmcraft-20221209-145322.062198.log'
➜  filebeat-charm git:(lma-improvements/graylog-tls) ✗ vi charmcraft.yaml
➜  filebeat-charm git:(lma-improvements/graylog-tls) ✗ charmcraft version
2.1.0

VariableDeclared · 2022-12-09T15:02:12Z

Following up on my last comment - the issue is related to the charm plugin and reactive plugin. In this case we're using reactive which doesn't appear to support this arguement.

VariableDeclared · 2022-12-12T16:44:29Z

metadata.yaml

+subordinate: true
+tags:
+ - filebeat
+requires:
+ beats-host:
+ interface: juju-info
+ scope: container
+ logstash:
+ interface: elastic-beats
+ elasticsearch:
+ interface: elasticsearch
+ kafka:
+ interface: kafka


hey @esunar and @agileshaw I've pushed some changes which I hope will help with the issues seen. Seems that build no longer works on my workstation, I had to add these lines to have a successful build.

The changes looks good to me. Could you please attach the log from some simple bundle deployment? (e.g. https://git.launchpad.net/charm-graylog/tree/src/tests/functional/tests/bundles/base-graylog.yaml). I would like to see all if this jammy-built charm doesn't break such a deployment.

hey sure, this is the bundle I am using today:

https://pastebin.canonical.com/p/kcddqf7WsQ/

And do you have a juju status (or something similar that shows deployment status) output of the deployed bundle?

sure, juju status is here: https://pastebin.canonical.com/p/jFRJTx3x7p/

VariableDeclared added 2 commits November 7, 2022 16:08

Add client TLS to Filebeat

15eb2ee

Added client TLS to filebeat to enable pushing to Graylog. TLS Client is only used to install the PKI CA and does not request certificates of its own.

Modified beats layer to accept additional config

226c6d9

VariableDeclared mentioned this pull request Nov 22, 2022

Enable extra parameters to be passed to the renderer canonical/layer-beats-base#42

Merged

PR Cleanup

63c667e

VariableDeclared changed the title ~~Lma improvements/graylog tls~~ Enable TLS in filebeat Nov 28, 2022

agileshaw reviewed Nov 28, 2022

View reviewed changes

agileshaw requested a review from esunar December 6, 2022 17:30

esunar approved these changes Dec 9, 2022

View reviewed changes

workaround charmcraft build issues

9099ca8

VariableDeclared commented Dec 12, 2022

View reviewed changes

remove unessesary series metadata change

756339e

esunar merged commit 4e3a715 into canonical:master Jan 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable TLS in filebeat #101

Enable TLS in filebeat #101

VariableDeclared commented Nov 22, 2022 •

edited

Loading

agileshaw left a comment

agileshaw Nov 28, 2022

VariableDeclared Nov 29, 2022

VariableDeclared Nov 29, 2022

agileshaw Nov 29, 2022

VariableDeclared Nov 30, 2022

VariableDeclared commented Dec 5, 2022

agileshaw commented Dec 5, 2022

VariableDeclared commented Dec 6, 2022

agileshaw commented Dec 6, 2022

lathiat commented Dec 9, 2022

VariableDeclared commented Dec 9, 2022

VariableDeclared commented Dec 9, 2022

VariableDeclared Dec 12, 2022

agileshaw Dec 12, 2022

VariableDeclared Dec 15, 2022

agileshaw Dec 15, 2022

VariableDeclared Jan 3, 2023

Enable TLS in filebeat #101

Enable TLS in filebeat #101

Conversation

VariableDeclared commented Nov 22, 2022 • edited Loading

agileshaw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VariableDeclared commented Dec 5, 2022

agileshaw commented Dec 5, 2022

VariableDeclared commented Dec 6, 2022

agileshaw commented Dec 6, 2022

lathiat commented Dec 9, 2022

VariableDeclared commented Dec 9, 2022

VariableDeclared commented Dec 9, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VariableDeclared commented Nov 22, 2022 •

edited

Loading