-
Notifications
You must be signed in to change notification settings - Fork 3.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cdk pipelines: Pipeline in infinite loop / always mutates and deploys #26471
Comments
Discovered the problem. Was tagging the stacks with a last modified timestamp. This obviously changed on every invocation. This was why it constantly cycled. Hopefully my error will help another dev solve their problem in the future. |
|
@miles-po thanks for sharing your research and I guess it may relate to my problem. My code pipeline is in a constant self-update mode when I this chunk of code handling env variables: for (const name of APP_STAGE_ENV_VARIABLES) {
const secretName: AppStageSecretName = `${stage}-${name}`;
envVariables[name] = cdk.SecretValue.secretsManager(secretName).toString();
} Do you think it relates? Thank you in advance for your help |
@Jurajzovinec If I had to guess, I'd say one of your environment variables keeps changing. Look for one with either a timestamp or a unique id like a build job id. |
Describe the bug
Seeing a serious issue where my CodePipeline always triggers a pipeline mutation. It continues building assets for each invocation but puts the Source and Builds steps in a loop. Something’s broken, but I don’t know where to look to unbreak it. I have deleted the stack and recreated, but it continues to happen. Something in the common CDK config? This started happening when I enabled caching in another branch, but I have since reverted the change. There is no caching on the pipeline anymore. There is no custom hashing or signature logic.
Expected Behavior
Once-through per push to the repository (perhaps two on an actual pipeline stack change when a mutation is warranted).
Current Behavior
Source ->
Build ->
UpdatePipeline -> Assets, etc.
Source ->
Build ->
UpdatePipeline -> Assets, etc.
Source ->
Build ->
UpdatePipeline -> Assets, etc.
Reproduction Steps
Don't know for sure how to reproduce. Perhaps overcache with custom and then turn off custom caching?
Possible Solution
No response
Additional Information/Context
The pipeline is not complex. GitHub source with token from SecretsManager.
npm ci
in the CDK directory andcdk synth
during build. In a monorepo where the CDK code is in acdk
subdirectory.cdk/cdk.out
is the primary output directory relative to repo root. Large ARM64 compute type for the build/synth step. Normal ARM64 instances for all other steps. One stage with an app stack. No direct filesystem calls. Nothing weird. Just CDK API.All build logs appear to be identical. No clear reason why a stack mutation would take place.
CDK CLI Version
2.87.0
Framework Version
No response
Node.js Version
18.16.1
OS
CodePipeline/CodeBuild's Linux
Language
Typescript
Language Version
~4.9.5
Other information
No response
The text was updated successfully, but these errors were encountered: