Error Elasticsearch_keystore[node0] Could not evaluate

[hoggle64]

• Module version: 6.3.3
• Puppet version: 5.5.19
• OS and version: CentOS Linux release 7.7.1908 (Core)

Bug description

When passing the following secrets hash to the instance:

  secrets:
    'xpack.security.authc.realms.active_directory.active_directory1.secure_bind_password': <%= @host.params['LDAP_BIND_PASSWORD'] %>

we receive the following error during the puppet agent run:

Debug: Augeas[defaults_node0](provider=augeas): Skipping because no files were changed
Debug: Augeas[defaults_node0](provider=augeas): Closed the augeas connection
Debug: Prefetching elasticsearch_keystore resources for elasticsearch_keystore
Notice: /Stage[main]/Elasticsearch_multinodes::Init/Elasticsearch::Instance[node0]/Elasticsearch_keystore[node0]/ensure: created (corrective)
Debug: Executing with uid=elasticsearch gid=elasticsearch: '/usr/share/elasticsearch/bin/elasticsearch-keystore create'
Error: /Stage[main]/Elasticsearch_multinodes::Init/Elasticsearch::Instance[node0]/Elasticsearch_keystore[node0]: Could not evaluate:
Debug: Prefetching ruby resources for elasticsearch_service_file
Debug: /Stage[main]/Elasticsearch_multinodes::Init/Elasticsearch::Instance[node0]/Elasticsearch::Service[node0]/Elasticsearch::Service::Systemd[node0]/Exec[systemd_reload_node0]: '/bin/systemctl daemon-ref failed check 'refreshonly'

I have no clue why this is failing.

I will attach the entire managing class.

init.pp.txt

[hoggle64]

Puppet Trace&Debug:

puppet_agent_debug_trace.txt

[hoggle64]

it might be related to the elasticsearch_keystore type:

def self.run_keystore(args, instance, configdir = '/etc/elasticsearch', stdin = nil)
  options = {
    :custom_environment => {
      'ES_INCLUDE' => File.join(defaults_dir, "elasticsearch-#{instance}"),
      'ES_PATH_CONF' => "#{configdir}/#{instance}"
    },
    :uid => 'elasticsearch',
    :gid => 'elasticsearch'
  }

I have added a debug output for the value "#{configdir}/#{instance}" and the value is:
/etc/elasticsearch/node0
but the correct path would be:
/etc/elasticsearch/ONE4ARCH/node0

[fatmcgav]

@hoggle64 It looks like you're on the right path...

configdir can be specified on the elasticsearch_keystore type, but it looks like elasticsearch::instance is defaulting the value to $elasticsearch::config:
https:/elastic/puppet-elasticsearch/blob/d6d3bb6b40b91cac5410a9cc406621c744f6c3a6/manifests/instance.pp#L502-L508

This is a bug in that it's not correctly handling a different configdir setup.

However, it's worth noting that multi-instance support is being removed in the next major version of this module. See #1068 for more info.

You can possibly work-around the current bug by managing the secrets directly in your main class, rather than relying on elasticsearch::instance to manage them.
Or patching the module as follows:

      elasticsearch_keystore { $name :
-       configdir => $elasticsearch::configdir,
+       configdir => $configdir,
        purge     => $purge_secrets,
        settings  => merge($main_secrets, $instance_secrets),
        notify    => $notify_service,

[hoggle64]

Thanks for your valuable answer @fatmcgav,
I've now investigated further and the problem at this point is a bit different.

The issue is that the keystore gets created every time even though the file is existing already. Why does the module think it has to create it ?

I retraced the steps manually and did the following:

[root@logte06011 ~]# sudo su - elasticsearch
-bash-4.2$ export ES_PATH_CONF=/etc/elasticsearch//ONE4ARCH/node0
-bash-4.2$ export "ES_INCLUDE"="/etc/sysconfig/elasticsearch-node0"
-bash-4.2$ /usr/share/elasticsearch/bin/elasticsearch-keystore create
An elasticsearch keystore already exists. Overwrite? [y/N]

Thanks in advance for your support.

[fatmcgav]

At a quick glance, I suspect there's an issue with the prefetch functionality, whereby it's only expecting a single level of node directories under /etc/elasticsearch:
https:/elastic/puppet-elasticsearch/blob/d6d3bb6b40b91cac5410a9cc406621c744f6c3a6/lib/puppet/provider/elasticsearch_keystore/elasticsearch_keystore.rb#L57-L69

Unfortunately I can't see an easy way around that, and as I've mentioned above, we are removing the multi instance functionality in the next major release.

[hoggle64]

Mhhh... got it.

Then I assume we will have to configure this in our own elastic module since we are running multiple clusters with multiple nodes on one machine.

Thank you for your time.

[fatmcgav]

Afraid so @hoggle64

since we are running multiple clusters with multiple nodes on one machine.

Out of curiosity, what's the use case behind this? As we (Elastic) have seen issues in the past with resource contention etc due to multiple instances on a single node.
And would Docker not be more suitable?

Edit: feel free to reach out via email if you'd prefer. Gavin.williamselastic.co

[hoggle64]

Hi @fatmcgav,

I fully agree. Unfortunately some companies refuse to use modern technologies like virtulization or containerization. Then you end up with bare metal machines having 1TB RAM which are setup and then running for years which also makes config-managment and provision-on-demand obsolete.

There are things you can't change....

[fatmcgav]

@hoggle64 thank you for the feedback...

I'm going to go ahead and close this issue out.