AWS Secrets Manager

[mtibben]

AWS has just announced Secrets Manager

• blog: https://aws.amazon.com/blogs/aws/aws-secrets-manager-store-distribute-and-rotate-credentials-securely/
• docs: https://docs.aws.amazon.com/secretsmanager/latest/apireference/Welcome.html

This doesn't completely replace chamber though - chamber has the advantage of being a tiny binary that can sit in your docker container.

Do you think Chamber should switch backends from Parameter Store to use the Secrets Manager API directly? Or do you think chamber should stick to Parameter Store, and perhaps a new chamber-equivalent tool should fill this space?

[djrodgerspryor]

As far as I can tell, secrets-manager is just a much more expensive ($0.4 per secret/month + some per-query charges) version of parameter-store (which is free).

The only new feature is automatic rotation, which works automatically for RDS DB credentials, but it's literally just a cronjob for all other kinds of secrets (ie. you need to write your own lambda, just like you do with parameter store).

There's no reason not to support it as an alternate store backend, but replacing parameter-store seems like a bad idea.

Although I'm sure it's built on-top of parameter-store internally, I just checked and you can't see secrets-manager secrets in parameter-store, so chamber would need to merge the secrets from the two APIs manually at read-time, or force the user to pick a service and only read from that.

[FernandoMiguel]

When I read the blog post I thought the same, and don't understand why it's so much more expensive than store parameter

[emilva]

While Parameter Store is a free service, they still charge you for KMS keys and other underlying services like CloudWatch. In fact, Secrets Manager might be cheaper than parameter store, depending on how you manage your parameters and keys.

[djrodgerspryor]

You still need to pay for KMS keys to use Secrets Manager; each secret is associated with a KMS key, just like with ParameterStore SecureStrings.

[jeznag]

This article argues that rate limiting is an issue with SSM: https://www.stackery.io/blog/serverless-secrets/

[nickatsegment]

@jeznag That's 100% true and we hit it often (see #79). It's the main motivation for the experimental s3 backend

[emilva]

AWS Secrets Manager have 700 request/sec for ReadSecret https://docs.aws.amazon.com/secretsmanager/latest/userguide/reference_limits.html#reference_limits_rates

[nickatsegment]

One downside of Secrets Manager: It doesn't look like there's any way to get a batch of secrets, like you can with Parameter Store paths. You could put all the secrets for a service in one Secret, but you're limited to 4 KiB (same as Parameter Store).

[emilva]

Secrets Manager allows you to use / in secret names, so it is possible to create a "path" per service.
Example: You can prefix all secrets for myService with myService/. Then you can use the ListSecrets operation to get complete list of all secrets, after that you call GetSecretValue on the ones that has the correct prefix.
In addition you would create a custom IAM read-only policy for that specific prefix resource: arn:aws:secretsmanager:<region>:<accountId>:secret:myService/*

[nickatsegment]

Right, but you still need to make 1 ListSecrets call + n GetSecretValue calls vs SSM's 1 (paginated) GetParametersByPath call.

[emilva]

Yes, and I think that will suffice in most cases. The n GetSecretValue calls has a rate limit of 700/s, while GetParametersByPath has an unknown (?) rate limit, and only 10 parameters max per page.

[mumoshu]

@nickatsegment Just my two cents but I believe we can reduce n GetSecretValue calls to basically 1 SSM GetParameters call. SSM PS' GetParameters does pass-through to secrets by prefixing param names with /aws/reference/secretsmanager/.

[nickatsegment]

@mumoshu That's pretty cool. Hard to say which would be more efficient in the general case, but it'd be nice if it were user configurable.

[danielgsims]

Has there been any more interest in adopting secrets manager as a backend store?

[AnthonyWC]

I think one of the best feature of Secret Manager is dynamic generated secret which means you don't actually have to store your secret anywhere except in the secret store itself. That is huge for CI/CD since it solves the issue of how to create and store secret as part of CI/CD by eliminating the need of storing it altogether.

[nickatsegment]

@danielgsims I don't think there's a use case at Segment, so I doubt we'd write one in the foreseeable future. I'd happily accept a nice PR.

[njflowers]

I was interested in using Secrets Manager in conjunction with chamber, so I went ahead and wrote an implementation of a new backend yesterday. It's working pretty well now, although I did run into some fundamental issues:

• Secrets Manager doesn't seem to expose last changed user anywhere (can't support user visibility in history or list)
• Versions seem to be automatically generated and alphanumeric, and it doesn't look like you can vary tags by version (can't support chronological version numbers, and version fields had to be refactored from integers to strings)

Any suggestions for the above issues? I'm looking into using the SSM API for fetching secrets, but it doesn't look like it will be able to address the problems above. I'm also worried about adding dependency on the SSM API, given that one major motivation for this backend is to avoid rate throttling.

[stale]

This issue has been automatically marked stale because it has not had any activity in the last 60 days. If no further activity occurs within 7 days, it will be closed. Closed does not mean "never", just that it has no momentum to get accomplished any time soon.
See CONTRIBUTING.md for more info.

[stale]

Closing due to staleness. Closed does not mean "never", just that it has no momentum to get accomplished any time soon.
See CONTRIBUTING.md for more info.

[vishcious]

@njflowers i was looking at implementing a secretsmanager store myself and ran into the same problems that you did. Where did you end up? Is your version of it on github? I am thinking about creating a version with only support for secretsmanager and not SSM given that the Store interface doesn't carry over well

[njflowers]

@viswaug Yes, this code is available on GitHub. Please feel free to view / use it:

https:/njflowers/chamber/tree/add-secrets-manager-backend

I wasn't able to get around the limitations I mentioned regarding user visibility and versions, although I did get version display working. Since I never got a verdict regarding SSM vs SM APIs, I decided to implement them both - you can configure the behavior used by the Secrets Manager backend with the CHAMBER_SM_USE_SSM environment variable (setting any value implies true). I believe most behavior is working in that branch, although it has not been rebased for some time.