Response limit of 4k for CloudFormation Custom Resources

[joshlartz]

Using the AwsCustomResource construct to easily pass in SDK calls is very useful. Some service calls however have a large response body and lead to Response object is too long. in their CloudFormation response.

It would be nice to have a conditional to check for that and truncate, or more ideally, have a property to control what output goes into the response body, similar to how physicalResourceIdPath works.

[jogold]

Can you detail your use case ?

[joshlartz]

I ran into the 4k Custom Resource response limit when creating ECS services. Namely, updates to ECS services return an events key that contains potentially quite a bit of data. I had to go back to my own Lambda to prune out the data before responding to CloudFormation.

Some way to automatically resolve that issue, or specify only what data I want to return to CloudFormation would help here.

[SimonLegg]

The AWS CLI provides a --query parameter for doing this - might be of use: Example in EC2 Describe Instances

Note: i'm suffering from the same issue.

[ilko-rbi]

Can the implementation be extended to support several output paths - currently it is startsWith check as far as I can see in the commit, which is quite limiting. Example describeDBInstances, where one needs to get several parameters. At the moment DBInstances.0 is already too long :-(

[jogold]

Can the implementation be extended to support several output paths - currently it is startsWith check as far as I can see in the commit, which is quite limiting. Example describeDBInstances, where one needs to get several parameters. At the moment DBInstances.0 is already too long :-(

Would outputPaths: string[] be a solution? How would you use it in your case? outputPaths: ['DBInstances.0.Xxx', 'DBInstances.1.Xxx']?

[ilko-rbi]

yes, this will perfectly suit me

[dacort]

FYI, I ran into a similar error message but it was due to an error in my helm chart where the resulting error message was too big for Lambda to return. Looking at the Lambda logs showed the specific error message.

[NukaCody]

This shamefully took me 11 attempts to get right so here's a working example to save others that come across this

    const cr = new AwsCustomResource(this, "EC2InstancePrivateIPv6", {
      onCreate: {
        service: "EC2",
        action: "describeInstances",
        physicalResourceId: PhysicalResourceId.of(`${db.instanceId}-ipv6`),
        parameters: {
          InstanceIds: [db.instanceId],
        },
        outputPaths: ["Reservations.0.Instances.0.NetworkInterfaces.0.Ipv6Addresses.0.Ipv6Address"],
      },
      policy: AwsCustomResourcePolicy.fromSdkCalls({ resources: AwsCustomResourcePolicy.ANY_RESOURCE }),
    });

    const ipv6Address = cr.getResponseField("Reservations.0.Instances.0.NetworkInterfaces.0.Ipv6Addresses.0.Ipv6Address");
    new AaaaRecord(this, "IPv6Record", {
      zone: privateZone,
      recordName: "pg",
      target: RecordTarget.fromIpAddresses(ipv6Address),
    });

It turned out I was assuming the response would be return like an API call Reservations[0].Instances[0]...etc. Double check the lambda logs if you run across the response limit error and see how the data Is outputed

[joshbalfour]

I'm facing this right now with CrossRegionExportWriter although to be fair the keys it's generating are 292 characters long 😬

[kamami]

I am getting the issue too when trying to deploy...

    const iotCreateKeysAndCertificateCr = new cr.AwsCustomResource(this, 'CreateKeysAndCertificate', {
      policy: cr.AwsCustomResourcePolicy.fromStatements([
        new iam.PolicyStatement({
          effect: iam.Effect.ALLOW,
          resources: cr.AwsCustomResourcePolicy.ANY_RESOURCE,
          actions: ['iot:CreateKeysAndCertificate', 'iot:UpdateCertificate'],
        }),
      ]),
      logRetention: logs.RetentionDays.ONE_DAY,
      onCreate: {
        service: 'Iot',
        action: 'createKeysAndCertificate',
        parameters: {
          setAsActive: true,
        },
        physicalResourceId: cr.PhysicalResourceId.fromResponse('certificateId'),
      },
      onDelete: {
        service: 'Iot',
        action: 'updateCertificate',
        parameters: {
          certificateId: new cr.PhysicalResourceIdReference(),
          newStatus: 'INACTIVE',
        },
      },
    });

[amonigal]

Can someone explain the solution differently? I don't see where the fix is... how do you see the response size? How do you know what to enter for outputPaths value?

// File: cdk/lambdas/init-database.ts
export async function handler(event: any, context: Context): Promise<any> {
	const db_username = process.env.PGUSER;
	const db_password = process.env.PGPASSWORD;
	const db_host = process.env.PGHOST;
	const db_port = process.env.PGPORT;
	const db_name = process.env.db_name;
	console.log('DB_USERNAME: ', db_username);
	console.log('DB_HOST: ', db_host);
	console.log('DB_PORT: ', db_port);
	console.log('DB_NAME: ', db_name);
	const client = new Client();
	await client.connect();
	if (event.params.operation === 'create') {
		const res = await client.query(
            `CREATE database ${db_name}`
		);
		console.log(res);
	} else if (event.params.operation === 'delete') {
		const res = await client.query(
            `DROP database ${db_name}`
		);
		console.log(res);
	}
	return null;
}

// File: Stack file
		const fn = new NodejsFunction(this, 'DatabaseFn', {
			allowAllOutbound: true,
			bundling: {
				format: OutputFormat.ESM,
			  },
			entry: 'cdk/lambdas/init-database.ts',
			environment: {
				PGUSER: lProps.db_username,
				PGPASSWORD: lProps.db_password,
				PGHOST: lProps.db_host,
				PGPORT: `${lProps.db_port}`,
				db_name: lProps.db_name
			},
			functionName: `db-init-${lProps.customer}-${lProps.db_name}`,
			logRetention: RetentionDays.FIVE_MONTHS,
			runtime: Runtime.NODEJS_18_X,
			securityGroups: lProps.fnSecurityGroups,
			timeout: Duration.minutes(2),
			vpc: lProps.vpc,
			vpcSubnets: lProps.subnetsSelection,
		});

		const payloadCreate: string = JSON.stringify({
			params: {
				operation: 'create'
			}
		});
		
		const sdkCallCreate: AwsSdkCall = {
			service: 'Lambda',
			action: 'invoke',
			parameters: {
				FunctionName: fn.functionName,
				Payload: payloadCreate
			},
			physicalResourceId: PhysicalResourceId.of(Date.now().toString())
		};
		
		new AwsCustomResource(this, 'AwsCustomResource', {
			installLatestAwsSdk: false,
			onUpdate: sdkCallCreate,
			timeout: Duration.minutes(10),
			role: lProps.role,
			policy: AwsCustomResourcePolicy.fromSdkCalls({
				resources: AwsCustomResourcePolicy.ANY_RESOURCE
			})
		});

[sladg]

This shamefully took me 11 attempts to get right so here's a working example to save others that come across this

    const cr = new AwsCustomResource(this, "EC2InstancePrivateIPv6", {
      onCreate: {
        service: "EC2",
        action: "describeInstances",
        physicalResourceId: PhysicalResourceId.of(`${db.instanceId}-ipv6`),
        parameters: {
          InstanceIds: [db.instanceId],
        },
        outputPaths: ["Reservations.0.Instances.0.NetworkInterfaces.0.Ipv6Addresses.0.Ipv6Address"],
      },
      policy: AwsCustomResourcePolicy.fromSdkCalls({ resources: AwsCustomResourcePolicy.ANY_RESOURCE }),
    });

    const ipv6Address = cr.getResponseField("Reservations.0.Instances.0.NetworkInterfaces.0.Ipv6Addresses.0.Ipv6Address");
    new AaaaRecord(this, "IPv6Record", {
      zone: privateZone,
      recordName: "pg",
      target: RecordTarget.fromIpAddresses(ipv6Address),
    });

It turned out I was assuming the response would be return like an API call Reservations[0].Instances[0]...etc. Double check the lambda logs if you run across the response limit error and see how the data Is outputed

I'm in same boat here. Took me an evening of debugging as there is no easy way to console.log what CR provides as response. I tried to look, but did not find any issue related to lack of documentation around outputPaths :/