Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Encode API key as base64 in common code #18945

Merged
merged 9 commits into from
Jun 9, 2020

Conversation

ycombinator
Copy link
Contributor

@ycombinator ycombinator commented Jun 3, 2020

What does this PR do?

Encodes the API key supplied to the Elasticsearch client with base64 encoding right before setting it in the Authorization request header.

Why is it important?

The Elasticsearch client is shared by the Elasticsearch output as well as the Elasticsearch monitoring reporter. Rather than base64-encoding the API key in each of these places, it is better to base64-encode it within the client code itself.

In fact, prior to this PR, we were base64-encoding the key in the Elasticsearch output but not in the Elasticsearch monitoring reporter.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 3, 2020

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [ycombinator commented: jenkins, run tests please]

  • Start Time: 2020-06-08T16:53:48.520+0000

  • Duration: 84 min 1 sec

Test stats 🧪

Test Results
Failed 0
Passed 9317
Skipped 1574
Total 10891

Steps errors

Expand to view the steps failures

  • Name: Make -C generator/_templates/beat test
    • Description: make -C generator/_templates/beat test

    • Duration: 7 min 34 sec

    • Start Time: 2020-06-08T18:08:01.825+0000

    • log

@ycombinator ycombinator marked this pull request as ready for review June 3, 2020 16:53
@ycombinator ycombinator requested review from adriansr and urso June 3, 2020 16:54
@ycombinator
Copy link
Contributor Author

From my checking, the APM Server code is not directly calling eslegclient.NewConnection anywhere so they should be good with this change. Nevertheless, it would be good for someone from @elastic/apm-server to confirm. Thank you!

@ycombinator ycombinator added bug Team:Services (Deprecated) Label for the former Integrations-Services team v7.8.1 v7.9.0 v8.0.0 labels Jun 3, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-services (Team:Services)

@ycombinator ycombinator added [zube]: In Review needs_backport PR is waiting to be backported to other branches. labels Jun 3, 2020
Copy link
Contributor

@adriansr adriansr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the quick fix!

@simitt
Copy link
Contributor

simitt commented Jun 3, 2020

Thanks for the ping, the APM Server will not be affected by this change.

@@ -436,7 +437,8 @@ func (conn *Connection) execHTTPRequest(req *http.Request) (int, []byte, error)
}

if conn.APIKey != "" {
req.Header.Add("Authorization", "ApiKey "+conn.APIKey)
encoded := base64.StdEncoding.EncodeToString([]byte(conn.APIKey))
req.Header.Add("Authorization", "ApiKey "+encoded)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would it make sense to cache the result in a private variable like conn.encodedAPIKey ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it would. No need to encode it each time we make a request to ES. Thanks for the suggestion, will implement!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in c3e5b43.

@@ -75,6 +76,8 @@ type ConnectionSettings struct {

Timeout time.Duration
IdleConnTimeout time.Duration

encodedAPIKey string // Base64-encoded API key
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we want to move the field to Connection ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I guess because you want to keep ConnectionSettings just as the public settings API?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in dc4e921.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I guess because you want to keep ConnectionSettings just as the public settings API?

yep.

@andrewkroh
Copy link
Member

andrewkroh commented Jun 4, 2020

On a related not I think our Beat docs are not clear on the expected format for the api_key value. I think the api_key reference docs should mention that the value is the "id and api_key joined by a colon". And this example is not correct since the value should be of the format '<id>:<api_key>'. Or am I misunderstanding how these are used?

From elasticsearch output:

Screen Shot 2020-06-04 at 5 59 59 PM

@ycombinator
Copy link
Contributor Author

Thanks for pointing that out, @andrewkroh. I agree that the example value in the doc is incorrect (it looks already base64-encoded). I will add the doc update to this PR.

@ycombinator
Copy link
Contributor Author

ycombinator commented Jun 4, 2020

Documentation for api_key setting updated in 22a25cc6be1c8f0c955f7a61a5c3f3503138628a. I deliberately used the same values and similar language from https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html. @andrewkroh @dedemorton would you mind taking a look? Thanks!

@@ -37,13 +37,14 @@ output.elasticsearch:
password: "{pwd}"
------------------------------------------------------------------------------

To use an API key to connect to {es}, use `api_key`.
To use an API key to connect to {es}, use `api_key`. The value must be the ID of
the API key and the API key joined by a colon.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This info would be good down on line 137 with the api_key.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 42d288616.

if conn.APIKey != "" {
req.Header.Add("Authorization", "ApiKey "+conn.APIKey)
if conn.encodedAPIKey != "" {
req.Header.Add("Authorization", "ApiKey "+conn.encodedAPIKey)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: We still have an unnecessary alloc + copy here. If we initialize encodedAPIKey = "ApiKey " + ..., then we can just pass it to Header as is.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed in 47c9aacaaa04e9cc1e304d517144386a77483e02.

@ycombinator ycombinator merged commit 3135985 into elastic:master Jun 9, 2020
@ycombinator ycombinator deleted the lb-mon-es-b64 branch June 9, 2020 08:42
ycombinator added a commit that referenced this pull request Jun 9, 2020
* Encode API key as base64 in common code

* Adding comment on API key field

* Adding CHANGELOG entries

* Adding test

* Base64-encode API key in constructor

* Move encodedAPIKey field to Connection

* Update doc on `api_key` setting value

* Adding API key format to setting section

* Compute entire API key auth header value up front
ycombinator added a commit that referenced this pull request Jun 23, 2020
* Encode API key as base64 in common code

* Adding comment on API key field

* Adding CHANGELOG entries

* Adding test

* Base64-encode API key in constructor

* Move encodedAPIKey field to Connection

* Update doc on `api_key` setting value

* Adding API key format to setting section

* Compute entire API key auth header value up front
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
* Encode API key as base64 in common code

* Adding comment on API key field

* Adding CHANGELOG entries

* Adding test

* Base64-encode API key in constructor

* Move encodedAPIKey field to Connection

* Update doc on `api_key` setting value

* Adding API key format to setting section

* Compute entire API key auth header value up front
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug needs_backport PR is waiting to be backported to other branches. Team:Services (Deprecated) Label for the former Integrations-Services team v7.8.1 v7.9.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

monitoring.elasticsearch.api_key needs to be base64 encoded
7 participants