Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(aws-cdk-lib/pipelines): (muliple s3 sources from same bucket fails) #19875

Open
mariusingjer opened this issue Apr 12, 2022 · 1 comment
Open
Labels
@aws-cdk/aws-s3 Related to Amazon S3 @aws-cdk/pipelines CDK Pipelines library bug This issue is a bug. effort/small Small work item – less than a day of effort p2

Comments

@mariusingjer
Copy link

mariusingjer commented Apr 12, 2022

Describe the bug

I add two files from the same s3 bucket as source:

    new pipelines.CodePipeline(this, 'pipeline', {
      synth: new pipelines.ShellStep('Build-codepipeline-and-self-update', {
        input: pipelines.CodePipelineSource.s3(
          artifactsBucket,
          'some-artifact.zip', {
          trigger: cdk.aws_codepipeline_actions.S3Trigger.NONE,
          actionName: 'source-nr-1'
        }),
        additionalInputs: {
          'somewhere': pipelines.CodePipelineSource.s3(
            artifactsBucket,
            'another-key-same-bucket.zip', {
            trigger: cdk.aws_codepipeline_actions.S3Trigger.NONE,
            actionName: 'source-nr-2'
          }),
        },
        commands: [
          '...',
        ],
      }),
    });

Fails with:

Error: Node with duplicate id: c8e9dd64d043805dea2bbc5f5d6ff6c442e1330971
    at Graph.add (C:\Kildekode\github\bsf-innsikt\deploy\service\node_modules\aws-cdk-lib\pipelines\lib\helpers-internal\graph.ts:180:15)
    at PipelineGraph.addAndRecurse (C:\Kildekode\github\bsf-innsikt\deploy\service\node_modules\aws-cdk-lib\pipelines\lib\helpers-internal\pipeline-graph.ts:266:12)
    at PipelineGraph.addAndRecurse (C:\Kildekode\github\bsf-innsikt\deploy\service\node_modules\aws-cdk-lib\pipelines\lib\helpers-internal\pipeline-graph.ts:270:33)
    at PipelineGraph.addBuildStep (C:\Kildekode\github\bsf-innsikt\deploy\service\node_modules\aws-cdk-lib\pipelines\lib\helpers-internal\pipeline-graph.ts:114:17)
    at new PipelineGraph (C:\Kildekode\github\bsf-innsikt\deploy\service\node_modules\aws-cdk-lib\pipelines\lib\helpers-internal\pipeline-graph.ts:74:29)
    at CodePipeline.doBuildPipeline (C:\Kildekode\github\bsf-innsikt\deploy\service\node_modules\aws-cdk-lib\pipelines\lib\codepipeline\codepipeline.ts:371:25)
    at CodePipeline.buildPipeline (C:\Kildekode\github\bsf-innsikt\deploy\service\node_modules\aws-cdk-lib\pipelines\lib\main\pipeline-base.ts:120:10)
    at new CodePipelineStack (C:\Kildekode\github\bsf-innsikt\deploy\service\lib\deploy-everything\code-pipeline-stack.ts:228:21)
    at main (C:\Kildekode\github\bsf-innsikt\deploy\service\bin\pipeline.ts:52:5)
    at Object.<anonymous> (C:\Kildekode\github\bsf-innsikt\deploy\service\bin\pipeline.ts:61:1)

Expected Behavior

Should not fail

Current Behavior

Throws error

Reproduction Steps

See above

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.20.0

Framework Version

No response

Node.js Version

16.14.2

OS

Windows 10

Language

Typescript

Language Version

No response

Other information

No response

@mariusingjer mariusingjer added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Apr 12, 2022
@github-actions github-actions bot added the @aws-cdk/aws-s3 Related to Amazon S3 label Apr 12, 2022
@mariusingjer
Copy link
Author

mariusingjer commented Apr 12, 2022

A workaround is to import the same source bucket twice in the stack.

    new pipelines.CodePipeline(this, 'pipeline', {
      synth: new pipelines.ShellStep('Build-codepipeline-and-self-update', {
        input: pipelines.CodePipelineSource.s3(
          bucket: // import bucket first time here,
          'some-artifact.zip', {
          trigger: cdk.aws_codepipeline_actions.S3Trigger.NONE,
          actionName: 'source-nr-1'
        }),
        additionalInputs: {
          'somewhere': pipelines.CodePipelineSource.s3(
            bucket: // import bucket second time here,
            'another-key-same-bucket.zip', {
            trigger: cdk.aws_codepipeline_actions.S3Trigger.NONE,
            actionName: 'source-nr-2'
          }),
        },
        commands: [
          '...',
        ],
      }),
    });

@ryparker ryparker added p2 effort/small Small work item – less than a day of effort p1 and removed p2 labels Apr 26, 2022
@peterwoodworth peterwoodworth removed the needs-triage This issue or PR still needs to be triaged. label Apr 27, 2022
@otaviomacedo otaviomacedo removed their assignment Jun 27, 2022
@comcalvi comcalvi added p2 and removed p1 labels Jan 27, 2023
@scanlonp scanlonp added the @aws-cdk/pipelines CDK Pipelines library label Oct 11, 2023
mergify bot pushed a commit that referenced this issue Oct 20, 2023
… in a pipeline causes duplicate id error (#27602)

Differentiates between sources of the same repository by appending the branch name onto the node id and input/output artifacts. This avoids the duplicate id errors for different branches of the same repository, as well as validating that each source is a unique repository & branch combination.

The only change to the CFN template is the input & output artifacts, but since these are not stateful resources, they can be modified without breaking changes. The artifacts are also updates in tandem, so the pipeline source behavior will stay the same.

This change impacts these `CodePipelineSource`s:
- `s3()` - `objectKey` appended
- `connection()`
- `codeCommit()`

Does not change `ecr()` behavior as there is no notion of folders or branches.
Does not change `github()` behavior as doing so would cause destructive changes of webhooks.

Closes #23916 and #19875.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-s3 Related to Amazon S3 @aws-cdk/pipelines CDK Pipelines library bug This issue is a bug. effort/small Small work item – less than a day of effort p2
Projects
None yet
Development

No branches or pull requests

6 participants