core: CrossRegionReferences does not work when deploying child stack in multiple regions that include the region of the parent stack.

[aagarwallazs]

Describe the bug

When trying deploy the same child stack in multiple regions where one of the region is same as the parent stack, that refers to one parent stack, the deployment of subsequent child stack fails while trying to update the exports of the parent stack.

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

When trying to deploy the same child stack in multiple regions (where one region is same as the parent's stack region), referring to one parent stack, the exports between the other stacks should not be deleted.

Current Behavior

AWS CDK is trying to delete the exports between two different stacks, which might be referring to one common stack output.

Example Infra : Deploy the dynamodb DB stack in us-east-1, which is the parent stack. This DB table details need to be passed to the child lambda stacks.

Let's take the following scenarios for the above example :

1. Deploy DynamoDb stack in us-east-1 --> Deploy lambda stack in us-east-1 --> Deploy lambda stack in eu-central-1. Deployment of lambda stack in eu-central-1 fails with the following error

dynamodb-stack | 0/5 | 7:08:56 AM | UPDATE_IN_PROGRESS   | AWS::CloudFormation::Stack      | dynamodb-stack User Initiated
dynamodb-stack | 0/5 | 7:08:58 AM | UPDATE_ROLLBACK_IN_P | AWS::CloudFormation::Stack      | dynamodb-stack Export dynamodb-stack:ExportsOutputRefdemotable002BE91A2220D30B cannot be deleted as it is in use by lambda-stack
dynamodb-stack | 1/5 | 7:09:00 AM | UPDATE_ROLLBACK_COMP | AWS::CloudFormation::Stack      | dynamodb-stack
dynamodb-stack | 2/5 | 7:09:01 AM | UPDATE_ROLLBACK_COMP | AWS::CloudFormation::Stack      | dynamodb-stack
[07:09:12] [AWS cloudformation 200 1.295s 0 retries] describeStackEvents({ StackName: 'dynamodb-stack', NextToken: undefined })

From the above, it looks like cdk is first trying to delete the local export for the same region (us-east-1) and then trying to create a cross regional export. But since there is lambda referring to the local export, it fails to delete the export.

2. Deploy DynamoDb stack in us-east-1 --> Deploy lambda stack in eu-central-1 --> Deploy lambda stack in us-east-1. Deployment of lambda stack in us-east-1 fails with the following error

dynamodb-stack | 0/5 | 4:52:59 PM | UPDATE_IN_PROGRESS   | AWS::CloudFormation::Stack | dynamodb-stack User Initiated
dynamodb-stack | 1/5 | 4:53:03 PM | UPDATE_COMPLETE_CLEA | AWS::CloudFormation::Stack | dynamodb-stack
dynamodb-stack | 1/5 | 4:53:05 PM | DELETE_IN_PROGRESS   | AWS::CloudFormation::CustomResource | ExportsWriterapsouth1D5FC4E14CB72EBD5
[16:53:11] [AWS cloudformation 200 1.009s 0 retries] describeStacks({ StackName: 'dynamodb-stack' })
[16:53:11] Stack dynamodb-stack has an ongoing operation in progress and is not stable (UPDATE_COMPLETE_CLEANUP_IN_PROGRESS)
[16:53:14] [AWS cloudformation 200 1.431s 0 retries] describeStackEvents({ StackName: 'dynamodb-stack', NextToken: undefined })
dynamodb-stack | 1/5 | 4:53:11 PM | DELETE_FAILED        | AWS::CloudFormation::CustomResource | ExportsWriterapsouth1D5FC4E14CB72EBD5 Received response status [FAILED] from custom resource. Message returned: Error: Exports cannot be updated:
/cdk/exports/lambda-stack/dynamodbstackuseast1FnGetAttdemotable002BE91AArn46AC3354 is in use by stack(s) lambda-stack
/cdk/exports/lambda-stack/dynamodbstackuseast1Refdemotable002BE91AD79CDF06 is in use by stack(s) lambda-stack
    at i (/var/task/index.js:4:10)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async h (/var/task/index.js:3:184)
    at async Runtime.handler (/var/task/__entrypoint__.js:1:932) (RequestId: 2faf739d-4cc5-49ec-a08d-da085bd65181)

From the above, it looks like cdk is first trying to delete the cross regional export (SSM parameter) for the eu-central-1 and then trying to create a local export. But since there is lambda referring to the cross regional export, it fails to delete the export.

Reproduction Steps

Repro the issue in 2 ways

1. Deploy DynamoDb stack in us-east-1 --> Deploy lambda stack in us-east-1 --> Deploy lambda stack in eu-central-1.
2. Deploy DynamoDb stack in us-east-1 --> Deploy lambda stack in eu-central-1 --> Deploy lambda stack in us-east-1.

Following code for the above example infra.

class DynamodbStack(Stack):

  def __init__(self, scope: Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        # create dynamo table
        demo_table = aws_dynamodb.Table(
            self, "demo_table",
            table_name=PhysicalName.GENERATE_IF_NEEDED,
            partition_key=aws_dynamodb.Attribute(
                name="id",
                type=aws_dynamodb.AttributeType.STRING
            ),
            removal_policy=RemovalPolicy.DESTROY
        )

        self.params = dict()
        self.params["DYNAMO_DB_TABLE"] = demo_table


class LambdaStack(Stack):
  def __init__(self, scope: Construct, id: str, params: dict, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        # create dynamo table
        demo_table = params["DYNAMO_DB_TABLE"]
        # create producer lambda function
        producer_lambda = aws_lambda.Function(self, "producer_lambda_function",
                                              runtime=aws_lambda.Runtime.PYTHON_3_10,
                                              handler="lambda_function.lambda_handler",
                                              code=aws_lambda.Code.from_asset("./lambda/producer"))

        producer_lambda.add_environment("TABLE_NAME", demo_table.table_name)

        # grant permission to lambda to write to demo table
        demo_table.grant_write_data(producer_lambda)


from dynamodb_lambda.dynamodb_lambda_stack import DynamodbStack
from dynamodb_lambda.lambda_stack import LambdaStack

app = App()
cdk_env_role=cdk.Environment(account="<>", region='us-east-1')


db_stack = DynamodbStack(app, "dynamodb-stack", env=cdk_env_role)

cdk_env_role_lambda=cdk.Environment(account="<>", region=os.environ.get("AWS_DEFAULT_REGION"))
if os.environ.get("AWS_DEFAULT_REGION") == "us-east-1":
    LambdaStack(app, "lambda-stack", db_stack.params, env=cdk_env_role_lambda)
else:
    LambdaStack(app, "lambda-stack", db_stack.params, env=cdk_env_role_lambda, cross_region_references=True)
app.synth()

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.140.0 (build 46168aa)

Framework Version

No response

Node.js Version

v20.13.0

OS

Windows Subsystem on Linux (Ubuntu 22.04)

Language

Python

Language Version

3.10.12

Other information

No response

[pahud]

Sounds like you are having multiple consumer stacks sharing a single producer stack?

As crossStackReference only supports strong reference, it would be a challenge with multiple consumer stacks sharing a single producer stack unless we support weak references. See Constraints in the doc for more details.

Making this a p2 FR and we welcome upvotes 👍 to help us prioritize.