Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add DynamoDB provider to parameters module #1091

Merged
merged 12 commits into from
Mar 20, 2023
62 changes: 55 additions & 7 deletions docs/utilities/parameters.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,9 @@ description: Utility


The parameters utility provides a way to retrieve parameter values from
[AWS Systems Manager Parameter Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html) or
[AWS Secrets Manager](https://aws.amazon.com/secrets-manager/). It also provides a base class to create your parameter provider implementation.
[AWS Systems Manager Parameter Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html),
[AWS Secrets Manager](https://aws.amazon.com/secrets-manager/), or [Amazon DynamoDB](https://aws.amazon.com/dynamodb/).
It also provides a base class to create your parameter provider implementation.

**Key features**

Expand Down Expand Up @@ -40,11 +41,12 @@ To install this utility, add the following dependency to your project.

This utility requires additional permissions to work as expected. See the table below:

Provider | Function/Method | IAM Permission
------------------------------------------------- | ------------------------------------------------- | ---------------------------------------------------------------------------------
SSM Parameter Store | `SSMProvider.get(String)` `SSMProvider.get(String, Class)` | `ssm:GetParameter`
SSM Parameter Store | `SSMProvider.getMultiple(String)` | `ssm:GetParametersByPath`
Secrets Manager | `SecretsProvider.get(String)` `SecretsProvider.get(String, Class)` | `secretsmanager:GetSecretValue`
Provider | Function/Method | IAM Permission
------------------------------------------------- |----------------------------------------------------------------------| ---------------------------------------------------------------------------------
SSM Parameter Store | `SSMProvider.get(String)` `SSMProvider.get(String, Class)` | `ssm:GetParameter`
SSM Parameter Store | `SSMProvider.getMultiple(String)` | `ssm:GetParametersByPath`
Secrets Manager | `SecretsProvider.get(String)` `SecretsProvider.get(String, Class)` | `secretsmanager:GetSecretValue`
DynamoDB | `DynamoDBProvider.get(String)` `DynamoDBProvider.getMultiple(string)` | `dynamodb:GetItem` `dynamoDB:Query`

## SSM Parameter Store

Expand Down Expand Up @@ -166,6 +168,52 @@ in order to get data from other regions or use specific credentials.
}
```

## DynamoDB
To get secrets stored in DynamoDB, use `getDynamoDbProvider`, providing the name of the table that
contains the secrets. As with the other providers, an overloaded methods allows you to retrieve
a `DynamoDbProvider` providing a client if you need to configure it yourself.

=== "DynamoDbProvider"

```java hl_lines="6 9"
import software.amazon.lambda.powertools.parameters.DynamoDbProvider;
import software.amazon.lambda.powertools.parameters.ParamManager;

public class AppWithDynamoDbParameters implements RequestHandler<APIGatewayProxyRequestEvent, APIGatewayProxyResponseEvent> {
// Get an instance of the DynamoDbProvider
DynamoDbProvider ddbProvider = ParamManager.getDynamoDbProvider("my-parameters-table");

// Retrieve a single parameter
String value = ddbProvider.get("my-key");
}
```

=== "DynamoDbProvider with an explicit region"
scottgerring marked this conversation as resolved.
Show resolved Hide resolved

```java hl_lines="9 10 11 12 15 18"
import software.amazon.lambda.powertools.parameters.DynamoDbProvider;
import software.amazon.lambda.powertools.parameters.ParamManager;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
import software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient;
import software.amazon.awssdk.regions.Region;

public class AppWithDynamoDbParameters implements RequestHandler<APIGatewayProxyRequestEvent, APIGatewayProxyResponseEvent> {
// Get a DynamoDB Client with an explicit region
DynamoDbClient ddbClient = DynamoDbClient.builder()
.httpClientBuilder(UrlConnectionHttpClient.builder())
.region(Region.EU_CENTRAL_2)
.build();

// Get an instance of the DynamoDbProvider
DynamoDbProvider provider = ParamManager.getDynamoDbProvider(ddbClient, "test-table");

// Retrieve a single parameter
String value = ddbProvider.get("my-key");
}
```



## Advanced configuration

### Caching
Expand Down
4 changes: 4 additions & 0 deletions powertools-parameters/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,10 @@
<groupId>software.amazon.awssdk</groupId>
<artifactId>url-connection-client</artifactId>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
scottgerring marked this conversation as resolved.
Show resolved Hide resolved
<artifactId>dynamodb</artifactId>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,179 @@
package software.amazon.lambda.powertools.parameters;

import software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider;
import software.amazon.awssdk.core.SdkSystemSetting;
import software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
import software.amazon.awssdk.services.dynamodb.model.*;
import software.amazon.lambda.powertools.parameters.cache.CacheManager;
import software.amazon.lambda.powertools.parameters.transform.TransformationManager;

import java.util.Collections;
import java.util.Map;
import java.util.stream.Collectors;

/**
* Implements a {@link ParamProvider} on top of DynamoDB. The schema of the table
* is described in the Python powertools documentation.
*
* @see <a href="https://awslabs.github.io/aws-lambda-powertools-python/latest/utilities/parameters/#dynamodbprovider">Python DynamoDB provider</a>
*
*/
public class DynamoDbProvider extends BaseProvider {

private final DynamoDbClient client;
private final String tableName;

public DynamoDbProvider(CacheManager cacheManager, String tableName) {
this(cacheManager, DynamoDbClient.builder()
.httpClientBuilder(UrlConnectionHttpClient.builder())
.credentialsProvider(EnvironmentVariableCredentialsProvider.create())
.region(Region.of(System.getenv(SdkSystemSetting.AWS_REGION.environmentVariable())))
.build(),
tableName
);

}

DynamoDbProvider(CacheManager cacheManager, DynamoDbClient client, String tableName) {
jeromevdl marked this conversation as resolved.
Show resolved Hide resolved
super(cacheManager);
this.client = client;
this.tableName = tableName;
}

/**
* Return a single value from the DynamoDB parameter provider.
*
* @param key key of the parameter
* @return The value, if it exists, null if it doesn't. Throws if the row exists but doesn't match the schema.
*/
@Override
protected String getValue(String key) {
GetItemResponse resp = client.getItem(GetItemRequest.builder()
.tableName(tableName)
.key(Collections.singletonMap("id", AttributeValue.fromS(key)))
.attributesToGet("val")
.build());

// If we have an item at the key, we should be able to get a 'val' out of it. If not it's
// exceptional.
// If we don't have an item at the key, we should return null.
if (resp.hasItem() && !resp.item().values().isEmpty()) {
return resp.item().get("val").s();
}

return null;
}

/**
* Returns multiple values from the DynamoDB parameter provider.
*
* @param path Parameter store path
* @return All values matching the given path, and an empty map if none do. Throws if any records exist that don't match the schema.
*/
@Override
protected Map<String, String> getMultipleValues(String path) {

QueryResponse resp = client.query(QueryRequest.builder()
scottgerring marked this conversation as resolved.
Show resolved Hide resolved
.tableName(tableName)
.keyConditionExpression("id = :v_id")
.expressionAttributeValues(Collections.singletonMap(":v_id", AttributeValue.fromS(path)))
.build());

return resp
.items()
.stream()
.collect(
Collectors.toMap(
(i) -> i.get("sk").s(),
(i) -> i.get("val").s()));


}

/**
* Create a builder that can be used to configure and create a {@link DynamoDbProvider}.
*
* @return a new instance of {@link DynamoDbProvider.Builder}
*/
public static DynamoDbProvider.Builder builder() {
return new DynamoDbProvider.Builder();
}

static class Builder {
private DynamoDbClient client;
private String table;
private CacheManager cacheManager;
private TransformationManager transformationManager;

/**
* Create a {@link DynamoDbProvider} instance.
*
* @return a {@link DynamoDbProvider}
*/
public DynamoDbProvider build() {
if (cacheManager == null) {
throw new IllegalStateException("No CacheManager provided; please provide one");
}
if (table == null) {
throw new IllegalStateException("No DynamoDB table name provided; please provide one");
}
DynamoDbProvider provider;
if (client != null) {
provider = new DynamoDbProvider(cacheManager, client, table);
} else {
provider = new DynamoDbProvider(cacheManager, table);
}
if (transformationManager != null) {
provider.setTransformationManager(transformationManager);
}
return provider;
}

/**
* Set custom {@link DynamoDbClient} to pass to the {@link DynamoDbClient}. <br/>
* Use it if you want to customize the region or any other part of the client.
*
* @param client Custom client
* @return the builder to chain calls (eg. <pre>builder.withClient().build()</pre>)
*/
public DynamoDbProvider.Builder withClient(DynamoDbClient client) {
this.client = client;
return this;
}

/**
* <b>Mandatory</b>. Provide a CacheManager to the {@link DynamoDbProvider}
*
* @param cacheManager the manager that will handle the cache of parameters
* @return the builder to chain calls (eg. <pre>builder.withCacheManager().build()</pre>)
*/
public DynamoDbProvider.Builder withCacheManager(CacheManager cacheManager) {
this.cacheManager = cacheManager;
return this;
}

/**
* <b>Mandatory</b>. Provide a DynamoDB table to the {@link DynamoDbProvider}
*
* @param table the table that parameters will be retrieved from.
* @return the builder to chain calls (eg. <pre>builder.withTable().build()</pre>)
*/
public DynamoDbProvider.Builder withTable(String table) {
this.table = table;
return this;
}

/**
* Provide a transformationManager to the {@link DynamoDbProvider}
*
* @param transformationManager the manager that will handle transformation of parameters
* @return the builder to chain calls (eg. <pre>builder.withTransformationManager().build()</pre>)
*/
public DynamoDbProvider.Builder withTransformationManager(TransformationManager transformationManager) {
this.transformationManager = transformationManager;
return this;
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@

import software.amazon.awssdk.services.secretsmanager.SecretsManagerClient;
import software.amazon.awssdk.services.ssm.SsmClient;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
import software.amazon.lambda.powertools.parameters.cache.CacheManager;
import software.amazon.lambda.powertools.parameters.transform.TransformationManager;

Expand All @@ -36,8 +37,8 @@ public final class ParamManager {

/**
* Get a concrete implementation of {@link BaseProvider}.<br/>
* You can specify {@link SecretsProvider} or {@link SSMProvider} or create your custom provider
* by extending {@link BaseProvider} if you need to integrate with a different parameter store.
* You can specify {@link SecretsProvider}, {@link SSMProvider}, {@link DynamoDbProvider}, or create your
* custom provider by extending {@link BaseProvider} if you need to integrate with a different parameter store.
* @return a {@link SecretsProvider}
*/
public static <T extends BaseProvider> T getProvider(Class<T> providerClass) {
Expand Down Expand Up @@ -65,6 +66,21 @@ public static SSMProvider getSsmProvider() {
return getProvider(SSMProvider.class);
}

/**
* Get a {@link DynamoDbProvider} with default {@link DynamoDbClient} <br/>
* If you need to customize the region, or other part of the client, use {@link ParamManager#getDynamoDbProvider(DynamoDbClient, String)}
*/
public static DynamoDbProvider getDynamoDbProvider(String tableName) {
// Because we need a DDB table name to configure our client, we can't use
// ParamManager#getProvider. This means that we need to make sure we do the same stuff -
// set transformation manager and cache manager.
return DynamoDbProvider.builder()
.withCacheManager(cacheManager)
.withTable(tableName)
.withTransformationManager(transformationManager)
.build();
}

/**
* Get a {@link SecretsProvider} with your custom {@link SecretsManagerClient}.<br/>
* Use this to configure region or other part of the client. Use {@link ParamManager#getSsmProvider()} if you don't need this customization.
Expand All @@ -91,6 +107,20 @@ public static SSMProvider getSsmProvider(SsmClient client) {
.build());
}

/**
* Get a {@link DynamoDbProvider} with your custom {@link DynamoDbClient}.<br/>
* Use this to configure region or other part of the client. Use {@link ParamManager#getDynamoDbProvider(String)} )} if you don't need this customization.
* @return a {@link DynamoDbProvider}
*/
public static DynamoDbProvider getDynamoDbProvider(DynamoDbClient client, String table) {
return (DynamoDbProvider) providers.computeIfAbsent(DynamoDbProvider.class, (k) -> DynamoDbProvider.builder()
.withClient(client)
.withTable(table)
.withCacheManager(cacheManager)
.withTransformationManager(transformationManager)
.build());
}

public static CacheManager getCacheManager() {
return cacheManager;
}
Expand Down
Loading