[jaeger-v2] Refactor ElasticSearch/OpenSearch Storage Configurations To Align With OTEL

[mahadzaryab1]

Requirement

In this issue, we want to align the configuration for elastic search and open search storage with OTEL (part of #5229)

Problem

• All config fields are tagged with mapstructure such that they can be addressable from YAML via "good" names
• Configs implement Validate()
• Configs via YAML support the same defaults as in jaeger-v1
• Configs reuse standard elements of OTEL configs whenever possible, e.g. TLS, clientcfg, etc.
• Configuration looks "logical" and properly grouped, not flattened with long weird names
• We create a migration guide document with a table that lists v1 CLI flags and the corresponding v2 config fields

[mahadzaryab1]

@yurishkuro is a migration guide needed here?

[yurishkuro]

yes, certainly

[mahadzaryab1]

@yurishkuro this is the current configuration we have

type Configuration struct {
	Servers                  []string               `mapstructure:"server_urls" valid:"required,url"`
	RemoteReadClusters       []string               `mapstructure:"remote_read_clusters"`
	Username                 string                 `mapstructure:"username"`
	Password                 string                 `mapstructure:"password" json:"-"`
	TokenFilePath            string                 `mapstructure:"token_file"`
	PasswordFilePath         string                 `mapstructure:"password_file"`
	AllowTokenFromContext    bool                   `mapstructure:"-"`
	Sniffer                  bool                   `mapstructure:"sniffer"` // https:/olivere/elastic/wiki/Sniffing
	SnifferTLSEnabled        bool                   `mapstructure:"sniffer_tls_enabled"`
	MaxDocCount              int                    `mapstructure:"-"` // Defines maximum number of results to fetch from storage per query
	MaxSpanAge               time.Duration          `mapstructure:"-"` // configures the maximum lookback on span reads
	Timeout                  time.Duration          `mapstructure:"-"`
	BulkSize                 int                    `mapstructure:"-"`
	BulkWorkers              int                    `mapstructure:"-"`
	BulkActions              int                    `mapstructure:"-"`
	BulkFlushInterval        time.Duration          `mapstructure:"-"`
	Indices                  Indices                `mapstructure:"indices"`
	ServiceCacheTTL          time.Duration          `mapstructure:"service_cache_ttl"`
	AdaptiveSamplingLookback time.Duration          `mapstructure:"-"`
	Tags                     TagsAsFields           `mapstructure:"tags_as_fields"`
	Enabled                  bool                   `mapstructure:"-"`
	TLS                      configtls.ClientConfig `mapstructure:"tls"`
	UseReadWriteAliases      bool                   `mapstructure:"use_aliases"`
	CreateIndexTemplates     bool                   `mapstructure:"create_mappings"`
	UseILM                   bool                   `mapstructure:"use_ilm"`
	Version                  uint                   `mapstructure:"version"`
	LogLevel                 string                 `mapstructure:"log_level"`
	SendGetBodyAs            string                 `mapstructure:"send_get_body_as"`
}

Here are some groupings I see.

Sniffer

• Enabled
• TLS Enabled

Bulk Processing

• Size
• Num Workers
• Num Actions
• Flush Interval

Auth

Can be grouped under basic if we want to support more authentication types in the future

• Username
• Password
• PasswordFilePath

Do you agree with the groupings above? Do you see any other groupings?

[yurishkuro]

I wonder if we can get rid of these if we switch to data streams

UseReadWriteAliases
CreateIndexTemplates
UseILM

[yurishkuro]

SendGetBodyAs - what are possible values? Should it be an enum?

[mahadzaryab1]

SendGetBodyAs - what are possible values? Should it be an enum?

it looks to be all the http method types

// SetSendGetBodyAs specifies the HTTP method to use when sending a GET request
// with a body. It is GET by default.
func SetSendGetBodyAs(httpMethod string) ClientOptionFunc {
	return func(c *Client) error {
		c.sendGetBodyAs = httpMethod
		return nil
	}
}

[mahadzaryab1]

I wonder if we can get rid of these if we switch to data streams

UseReadWriteAliases CreateIndexTemplates UseILM

what are data streams in this context?

[tronda]

@mahadzaryab1 Amazon described the data stream feature in this video:
https://www.youtube.com/watch?v=Mm3GFTt8wMA

I think this comes from ElasticSearch originally so the concept is probably the same.

[mahadzaryab1]

@yurishkuro final PR is ready for review at #6090