Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(s3-deployment): deploy data with deploy-time values #18659

Merged
merged 19 commits into from
Feb 3, 2022

Conversation

eladb
Copy link
Contributor

@eladb eladb commented Jan 26, 2022

Allow deploying test-based content that can potentially include deploy-time values such as attributes of cloud resources.

Introduce a Source.data(objectKey, text) and Source.jsonData(objectKey, obj) where the data can naturally include deploy-time tokens such as references to resources (Ref) or to resource attributes (Fn::GetAtt).

For example:

const appConfig = {
  topic_arn: topic.topicArn,
  base_url: 'https://my-endpoint',
};

new s3deploy.BucketDeployment(this, 'BucketDeployment', {
  sources: [s3deploy.Source.jsonData('config.json', config)],
  destinationBucket: destinationBucket,
});

This is implemented by replacing the deploy-time tokens with markers that are replaced inside the s3-deployment custom resource.

Fixes #12903


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

Allow deploying test-based content that can potentially include deploy-time values such as attributes of cloud resources.

Introduce a `Source.content(objectKey, text)` where `text` can naturally include tokens that resolve only at deploy time.
This is implemented by replacing the deploy-time tokens with markers that are replaced inside the s3-deployment custome resource.

Fixes #12903

TODO:

- [ ] Implement + test the custom resource code
- [ ] Documentation
- [ ] Finalize integration test
@gitpod-io
Copy link

gitpod-io bot commented Jan 26, 2022

@eladb eladb marked this pull request as draft January 26, 2022 09:58
@mergify mergify bot added the contribution/core This is a PR that came from AWS. label Jan 26, 2022
@jogold
Copy link
Contributor

jogold commented Jan 27, 2022

Hi @eladb,

FYI I'm currently addressing this like that:

new cr.AwsCustomResource(this, 'PutConfig', {
  policy: cr.AwsCustomResourcePolicy.fromStatements([
    new iam.PolicyStatement({
      actions: ['s3:PutObject'],
      resources: [props.bucket.arnForObjects('config.json')],
    }),
  ]),
  onUpdate: {
    service: 'S3',
    action: 'putObject',
    parameters: {
      Bucket: props.bucket.bucketName,
      Key: 'config.json',
      Body: Stack.of(this).toJsonString(props.config),
      ContentType: 'application/json',
      CacheControl: 'max-age=0, no-cache, no-store, must-revalidate',
    },
    physicalResourceId: cr.PhysicalResourceId.of('config'),
  },
});

@eladb
Copy link
Contributor Author

eladb commented Jan 27, 2022

@jogold this is very cool, but I since this means the contents of the file is inlined in the template, I think the 1MB limitation of the template size might be a big concern.

@jogold
Copy link
Contributor

jogold commented Jan 28, 2022

this means the contents of the file is inlined in the template, I think the 1MB limitation of the template size might be a big concern.

This is correct but one could argue that most of the use cases for files with deploy-time values are small files. Anyway, it's always nice (and fun 😄) to have a generalized solution.

@rfestag
Copy link

rfestag commented Jan 28, 2022

Definitely interested in this feature as it is developed!

A pattern I've run into before that was particularly frustrating is static websites that require deploy-time parameters (say, Cognito user pool information or bucket names). I generally do this through an S3 deployment bucket with a Dockerfile to build the assets. Is this something that could integrate easily with that? Or would the pattern be to render this content separately to the target bucket?

@eladb eladb marked this pull request as ready for review February 1, 2022 18:09
@eladb eladb changed the title feat(s3-deployment): deploy content with deploy-time values [WIP] feat(s3-deployment): deploy data with deploy-time values Feb 1, 2022
@ckifer
Copy link
Contributor

ckifer commented Feb 2, 2022

+1 to this, thank you @eladb !

Looking for this solution right now and glad to see it in development, I use the exact pattern internally but built in is going to be fantastic!

@otaviomacedo otaviomacedo added the pr/do-not-merge This PR should not be merged at this time. label Feb 2, 2022
@eladb eladb removed the pr/do-not-merge This PR should not be merged at this time. label Feb 3, 2022
@mergify
Copy link
Contributor

mergify bot commented Feb 3, 2022

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject89A8053A-LhjRyN9kxr8o
  • Commit ID: f80dd01
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mergify mergify bot merged commit d40e332 into master Feb 3, 2022
@mergify mergify bot deleted the benisrae/bucket-deployment-content-source branch February 3, 2022 10:45
@mergify
Copy link
Contributor

mergify bot commented Feb 3, 2022

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

kornicameister added a commit to kornicameister/aws-cdk that referenced this pull request Feb 6, 2022
* origin/master: (27 commits)
  chore(eks): deprecate older versions of EKS (aws#18842)
  fix(tooling): update vscode devcontainer image (aws#18455)
  chore: npm-check-updates && yarn upgrade (aws#18832)
  chore(docs): Fix broken md links (aws#18384)
  chore(lambda-layer-awscli): install awscli with pip and requirements.txt (aws#18800)
  fix(aws-appsync): Strip unsupported characters from Lambda DataSource (aws#18765)
  feat(cfnspec): cloudformation spec v55.0.0 (aws#18827)
  docs(cfnspec): update CloudFormation documentation (aws#18826)
  chore(cxapi): plugin context provider limited by cx schema (aws#18709)
  feat(iotevents): add grant method to Input class (aws#18617)
  chore(cx-api): break circular dependencies (aws#18767)
  docs(core): clarify that `addOverride` does not change property casing (aws#18687)
  feat(s3-deployment): deploy data with deploy-time values (aws#18659)
  docs(cfnspec): update CloudFormation documentation (aws#18808)
  feat(cli): `cdk diff` works for Nested Stacks (aws#18207)
  docs(cfnspec): update CloudFormation documentation (aws#18783)
  chore(lambda-layer-awscli): add update mechanism for AWS CLI (aws#18780)
  chore(release): 1.143.0
  feat(fsx): add support for FSx Lustre Persistent_2 deployment type (aws#18626)
  feat(amplify): support performance mode in Branch (aws#18598)
  ...
TikiTDO pushed a commit to TikiTDO/aws-cdk that referenced this pull request Feb 21, 2022
Allow deploying test-based content that can potentially include deploy-time values such as attributes of cloud resources.

Introduce a `Source.data(objectKey, text)` and `Source.jsonData(objectKey, obj)` where the data can naturally include deploy-time tokens such as references to resources (`Ref`) or to resource attributes (`Fn::GetAtt`).

For example:

```ts
const appConfig = {
  topic_arn: topic.topicArn,
  base_url: 'https://my-endpoint',
};

new s3deploy.BucketDeployment(this, 'BucketDeployment', {
  sources: [s3deploy.Source.jsonData('config.json', config)],
  destinationBucket: destinationBucket,
});
```

This is implemented by replacing the deploy-time tokens with markers that are replaced inside the s3-deployment custom resource.

Fixes aws#12903

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@RichiCoder1 RichiCoder1 mentioned this pull request Mar 9, 2023
2 tasks
@lafriakh
Copy link

@eladb I tried to use it with REST Api url and I get:

Error [ValidationError]: Template format error: Unresolved resource dependencies [...] in the Resources block of the template

I'm doing something wrong here, because your changes should make that possible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-s3-deployment contribution/core This is a PR that came from AWS.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

(aws-s3-deployment): assets with deploy-time values
8 participants