aws-codepipeline-actions: variables with special characters break lambda invoke JSON

[dleavitt]

Describe the bug

A CodePipeline AWS Lambda Invoke action will not receive valid JSON if passed a variable from a CodeConnection action that has a space or other special character in it.

More specifically, if your git commit message has a newline (or quote, etc) in it and you pass it as a UserParameter to a LambdaInvokeAction action, your lambda will crash when it tries to deserialize it.

This is going to affect pretty much anyone who's trying to use a Github commit message in a Pipeline like this, since Github's merge commit message have newlines in them.

This likely has the same root cause as #8458 but is maybe more egregious since LambdaInvokeAction specifically does JSON serialization in the CDK.

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

I'd expect the variable expansion to be JSON-safe or there to be some way to make it JSON-safe.

In general I'd expect output variables from actions to be usable as inputs to other actions, which currently doesn't seem to be the case.

Current Behavior

The variable expansion isn't valid JSON and as far as I know there's no way to get it to be. When you try to parse the UserParameters, your lambda will crash with a message like this:

Bad control character in string literal in JSON at position

Reproduction Steps

Create or find a git repo and connect it to AWS via a CodeConnection.

Make a couple of commits to the repo, one with a newline in the commit message, one without.

Deploy the stack below, passing it the arn of the code connection and the owner/name of the Git repo, like so:

CONNECTION_ARN=arn:aws:codeconnections:regions:account:connection/id GIT_REPO=owner/repo cdk deploy

Create a pipeline release of a commit where the commit message has a newline in the name. The release will fail.

import * as cdk from "aws-cdk-lib";
import { Construct } from "constructs";
import * as codepipeline from "aws-cdk-lib/aws-codepipeline";
import * as codepipeline_actions from "aws-cdk-lib/aws-codepipeline-actions";
import * as lambda from "aws-cdk-lib/aws-lambda";

interface ReproStackProps extends cdk.StackProps {
  connectionArn: string;
  repo: string;
}

class ReproStack extends cdk.Stack {
  constructor(
    scope: Construct,
    id: string,
    { connectionArn, repo, ...props }: ReproStackProps,
  ) {
    super(scope, id, props);

    const fn = new lambda.Function(this, "ActionLambda", {
      code: lambda.Code.fromInline(actionLambda),
      handler: "index.onEvent",
      runtime: lambda.Runtime.NODEJS_20_X,
    });

    const sourceOutput = new codepipeline.Artifact();

    const [repoOwner, repoName] = repo.split("/");

    const githubSourceAction =
      new codepipeline_actions.CodeStarConnectionsSourceAction({
        actionName: "GitSource",
        output: sourceOutput,
        connectionArn,
        owner: repoOwner,
        repo: repoName,
        // branch: "add-if-needed"
      });

    const pipeline = new codepipeline.Pipeline(this, "Pipeline", {
      stages: [
        {
          stageName: "Source",
          actions: [githubSourceAction],
        },
        {
          stageName: "Run",
          actions: [
            new codepipeline_actions.LambdaInvokeAction({
              actionName: "LambdaInvoke",
              lambda: fn,
              userParameters: {
                msg: githubSourceAction.variables.commitMessage,
                works: "a\nb",
              },
            }),
          ],
        },
      ],
    });
  }
}

const actionLambda = `
const { CodePipeline } = require("@aws-sdk/client-codepipeline");

async function onEvent(event, context, callback) {
  const codepipeline = new CodePipeline();

  const job = event["CodePipeline.job"];
  const payload = job.data.actionConfiguration.configuration.UserParameters;
  try {
    const r = JSON.parse(payload);
    console.log(r);
    await codepipeline.putJobSuccessResult({ jobId: job.id });
    callback(null);
  } catch (ex) {
    console.error(job.data.actionConfiguration.configuration);
    await codepipeline.putJobFailureResult({
      jobId: job.id,
      failureDetails: {
        type: "JobFailed",
        message: ex.message,
      },
    });
    callback(ex);
  }
}

module.exports = { onEvent };
`;

const connectionArn = process.env.CONNECTION_ARN;
const repo = process.env.GIT_REPO;

if (!repo || !connectionArn) {
  throw `Please pass GIT_REPO=org/repo and CONNECTION_ARN=arn:aws:codestar-connections:etc`;
}

const app = new cdk.App();
new ReproStack(app, "CdkReproStack", { connectionArn, repo });

Possible Solution

1. Add a way to get escaped values of action variables such that they can be used with other actions - #{XYZ.CommitMessage.json} or #{XYZ.quoted} or even #{XYZ.CommitMessageQuoted} or something. Doubt this is fixable at the CDK level.
2. The commitMessage variable especially seems like a footgun due to lack of escaping. Consider deprecating it or putting a warning in a docblock.

Additional Information/Context

No response

CDK CLI Version

2.150.0 (build 3f93027)

Framework Version

2.150.0

Node.js Version

20.11.1

OS

MacOS 14.3

Language

TypeScript

Language Version

TypeScript (5.5.3)

Other information

No response

[ashishdhingra]

Reproducible using customer provided code (for reproduction, new line character should be added when executing git commit -m " from console, pressing <<ENTER>> before closing message with " character).

If we tweak Lambda code to below (notice additional line console.log('PAYLOAD:\\n' + payload);):

const actionLambda = `
const { CodePipeline } = require("@aws-sdk/client-codepipeline");

async function onEvent(event, context, callback) {
  const codepipeline = new CodePipeline();

  const job = event["CodePipeline.job"];
  const payload = job.data.actionConfiguration.configuration.UserParameters;
  console.log('PAYLOAD:\\n' + payload);

  try {
    const r = JSON.parse(payload);
    console.log(r);
    await codepipeline.putJobSuccessResult({ jobId: job.id });
    callback(null);
  } catch (ex) {
    console.error(job.data.actionConfiguration.configuration);
    await codepipeline.putJobFailureResult({
      jobId: job.id,
      failureDetails: {
        type: "JobFailed",
        message: ex.message,
      },
    });
    callback(ex);
  }
}

module.exports = { onEvent };
`;

It logs below in CloudWatch when Lambda is executed:

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|   timestamp   |                                                                                                                                                                                                                                  message                                                                                                                                                                                                                                  |
|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1724696935703 | INIT_START Runtime Version: nodejs:20.v27 Runtime Version ARN: arn:aws:lambda:us-east-2::runtime:672d5a3e06f81d120c089c5414b05186d7b4098504797c766bde2459847f38bc                                                                                                                                                                                                                                                                                                         |
| 1724696936018 | START RequestId: 021a6b35-5b66-4a7d-958a-25a8ff60abd1 Version: $LATEST                                                                                                                                                                                                                                                                                                                                                                                                    |
| 1724696936089 | 2024-08-26T18:28:56.089Z 021a6b35-5b66-4a7d-958a-25a8ff60abd1 INFO PAYLOAD: {"msg":"Testing newline in commit message.","works":"a\nb"}                                                                                                                                                                                                                                                                                                                                   |
| 1724696936111 | 2024-08-26T18:28:56.111Z 021a6b35-5b66-4a7d-958a-25a8ff60abd1 ERROR {   FunctionName: 'CdkReproStack-ActionLambda1304E65B-waXCZ9u3WpLq',   UserParameters: '{"msg":"Testing newline in\ncommit message.","works":"a\\nb"}' }                                                                                                                                                                                                                                              |
| 1724696937169 | 2024-08-26T18:28:57.169Z 021a6b35-5b66-4a7d-958a-25a8ff60abd1 ERROR Invoke Error  {"errorType":"SyntaxError","errorMessage":"Bad control character in string literal in JSON at position 26","stack":["SyntaxError: Bad control character in string literal in JSON at position 26","    at JSON.parse (<anonymous>)","    at Runtime.onEvent [as handler] (/var/task/index.js:12:20)","    at Runtime.handleOnceNonStreaming (file:///var/runtime/index.mjs:1173:29)"]}  |
| 1724696937190 | END RequestId: 021a6b35-5b66-4a7d-958a-25a8ff60abd1                                                                                                                                                                                                                                                                                                                                                                                                                       |
| 1724696937190 | REPORT RequestId: 021a6b35-5b66-4a7d-958a-25a8ff60abd1 Duration: 1171.10 ms Billed Duration: 1172 ms Memory Size: 128 MB Max Memory Used: 83 MB Init Duration: 313.07 ms                                                                                                                                                                                                                                                                                                  |
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Notice that UserParamerters doesn't have escaped \n in msg parameter.

The synthesized template has AWS::CodePipeline::Pipeline resource as shown below:

...
"PipelineC660917D": {
   "Type": "AWS::CodePipeline::Pipeline",
   "Properties": {
    "ArtifactStore": {
     "Location": {
      "Ref": "PipelineArtifactsBucket22248F97"
     },
     "Type": "S3"
    },
    "PipelineType": "V2",
    "RoleArn": {
     "Fn::GetAtt": [
      "PipelineRoleD68726F7",
      "Arn"
     ]
    },
    "Stages": [
     {
      "Actions": [
       {
        "ActionTypeId": {
         "Category": "Source",
         "Owner": "AWS",
         "Provider": "CodeStarSourceConnection",
         "Version": "1"
        },
        "Configuration": {
         "ConnectionArn": "arn:aws:codestar-connections:us-east-2:139480602983:connection/629db161-8f01-4eb4-b763-77eef724298c",
         "FullRepositoryId": "ashishdhingra/testrepo",
         "BranchName": "master"
        },
        "Name": "GitSource",
        "Namespace": "Source_GitSource_NS",
        "OutputArtifacts": [
         {
          "Name": "Artifact_Source_GitSource"
         }
        ],
        "RoleArn": {
         "Fn::GetAtt": [
          "PipelineSourceGitSourceCodePipelineActionRole42BAD9EA",
          "Arn"
         ]
        },
        "RunOrder": 1
       }
      ],
      "Name": "Source"
     },
     {
      "Actions": [
       {
        "ActionTypeId": {
         "Category": "Invoke",
         "Owner": "AWS",
         "Provider": "Lambda",
         "Version": "1"
        },
        "Configuration": {
         "FunctionName": {
          "Ref": "ActionLambda1304E65B"
         },
         "UserParameters": "{\"msg\":\"#{Source_GitSource_NS.CommitMessage}\",\"works\":\"a\\nb\"}"
        },
        "Name": "LambdaInvoke",
        "RoleArn": {
         "Fn::GetAtt": [
          "PipelineRunLambdaInvokeCodePipelineActionRole851BD08E",
          "Arn"
         ]
        },
        "RunOrder": 1
       }
      ],
      "Name": "Run"
     }
    ]
   },
   "DependsOn": [
    "PipelineRoleDefaultPolicyC7A05455",
    "PipelineRoleD68726F7"
   ],
   "Metadata": {
    "aws:cdk:path": "CdkReproStack/Pipeline/Resource"
   }
  },
...

The UserParameters just has reference to #{Source_GitSource_NS.CommitMessage}. The expression is generated via variableExpression() helper method (also see CodePipeline: Variables reference).

@dleavitt Thanks for opening the issue. I came across the AWS post https://repost.aws/questions/QU2jfrtUlyQqyQ5YuUSgIEow/unexpected-codepipline-variable-resolution-error-bug and it appears that CodePipeline doesn't escape special characters at this moment. The only workaround is to preprocess variables individually in a Lambda. It's a CodePipeline limitation, not the CDK limitation, hence, we should not deprecate commitMessage property. Adding warning could be an option, but it could get out-of-sync in case CodePipeline decides to handle escape characters.

Please review above findings and confirm if we could close the issue.

Thanks,
Ashish

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[github-actions]

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.