Aurora Serverless v2

[autarchprinceps]

Name of the resource

AWS::RDS::DBInstance

Resource name

No response

Description

Aurora Serverless v2 is now GA. From how I understand the creation via UI/aws-cli the instances need to be of instance type db.serverless, which may or may not need to be added in CloudFormations property validation, but I definitively haven't found any way to specify the minimum and maximum ACUs yet. This is no longer a cluster setting, as with serverless v1, and even if, it now supports other values (e.g. 0.5) which definitively is neither documented as such or supported.

Other Details

https://aws.amazon.com/blogs/aws/amazon-aurora-serverless-v2-is-generally-available-instant-scaling-for-demanding-workloads/

[taspeotis]

I have used Cloudformation with db.serverless just fine - but there is no way to set min/max.

There is some suggestion from aws rds ... that ServerlessV2ScalingConfiguration exists for v2, replacing ScalingConfiguration.

[autarchprinceps]

@taspeotis I do not see, how you could have used CloudFormation just fine. It errors for me:
Instance:
Set the Serverless v2 scaling configuration on the parent DB cluster before creating a Serverless v2 DB instance.
Cluster:
Encountered unsupported property ServerlessV2ScalingConfiguration

I both need and cannot set the min & max values.

[taspeotis]

Again:

I have used Cloudformation with db.serverless just fine - but there is no way to set min/max.

You can configure the min and max in the web interface, and then use Cloudformation subsequently with the db.serverless engine version.

I did not say I could set ServerlessV2ScalingConfiguration. Again:

there is no way to set min/max.

[autarchprinceps]

"You can configure the min and max in the web interface" - only during initial cluster setup, which also creates the instances. Pressing modify on an existing DB Cluster without serverless instances, doesn't expose this setting, only modify to change the already set values, not setting it initially.
Also combination of manual & CloudFormation steps isn't the goal of IaC.
Clearly, ServerlessV2ScalingConfiguration needs to be added to CloudFormation DBCluster then, for Serverless v2 to be actually supported by CloudFormation. I expect, that then it should work, since db.serverless works for you on creating Instances. Apparently, there is an internal issue to add that, but no ETA as usual.

[taspeotis]

You can set them after the fact, modify one of the database instances and change it to serverless v2. And then you can run your cloudformation stack. I converted one instance to serverless v2 and that has the effect of setting min/max on the cluster. Then I submitted my cloudformation template and it set the rest of the instances to db.serverless.

I've migrated four clusters in place to serverless v2 with a combination of the web interface and cloudformation.

[cmesas]

I also have the problem to be able to define the configuration of the cluster. Any update with the resource structure to be able to specify the Max/Min ACUs in the cluster?

[kftsehk]

The closest thing now seem to be

1. Deploy AWS::RDS::DBCluster with EngineMode: provision
2. Run aws rds modify-db-cluster --serverless-v2-scaling-configuration '{"MinCapacity":0.5,"MaxCapacity":128}' --db-cluster-identifier <ClusterId>, if you have not find this config, update the awscli
3. Deploy AWS::RDS::DBInstance with DBInstanceClass: db.serverless

[autarchprinceps]

#1153

[dovidkopel]

+1

[arogozhnikov]

There are ~300 requests in CDK for serverless aurora (aws/aws-cdk#20197), issue was given p1, but it is blocked by lack of support in cloudformation

[driverpt]

any ETA on this one?

[bilalq]

This issue is more upvoted than the vote count would suggest. Like someone mentioned earlier in the comments, #1153 is also effectively an issue covering the same need (just referencing a different specific CloudFormation type) and there's the CDK issue at aws/aws-cdk#20197.

There are about 200 thumbs up here, about 100 in the other issue in the repo, and over 300 in the CDK one. I know there's likely to be some overlap in the thumbsup counts, but it's still a substantial ask from end customers.

I'm sure the CloudFormation team has heard this before, but it's really frustrating when a feature/service launches "GA" without CFN or CDK support.

[echennh-zz]

any ETA on this one?

@driverpt

Just heard back from AWS Support, and they said

Having said that, as you know any new feature or a fix requires the product to undergo rigorous testing and troubleshooting before it is made available, to ensure security and stability for end users. Unfortunately, I will not be able to provide you an ETA for it at the moment as the internal team does not share this information with us.

However, you also can track the Aurora Serverless v2 coverage via the CloudFormation GitHub Roadmap. There is currently an opened issue discussing this use case here [2][3].

[echennh-zz]

You can set them after the fact, modify one of the database instances and change it to serverless v2. And then you can run your cloudformation stack. I converted one instance to serverless v2 and that has the effect of setting min/max on the cluster. Then I submitted my cloudformation template and it set the rest of the instances to db.serverless.

I've migrated four clusters in place to serverless v2 with a combination of the web interface and cloudformation.

@taspeotis

Hi, can you clarify in more detail how you did this? Did you create the Aurora cluster in the console as a v2 cluster, then deploy a Cloudformation stack where one of the resources is the pre-existing cluster? If so, how did you do that?
Up to this point, I've tried to update a stack via a CloudFormation template, but obviously since CF doesn't support Aurora v2, it just creates a new database cluster of type Aurora v1.
Did you deploy a CF template and specify the SourceDBClusterIdentifier property or the SnapshotIdentifier to get it to look at the Aurora v2 cluster created in the console?

Any help appreciated!

[deuscapturus]

This hasn't even made it to the "Researching" phase of the roadmap. Why not? This issue seems to have fallen through the cracks. Could someone from AWS please provide a response? Thanks!

[gazal-k]

Interestingly, AWS Controllers for Kubernetes (ACK) already has support. https://aws-controllers-k8s.github.io/community/reference/rds/v1alpha1/dbcluster/ serverlessV2ScalingConfiguration. It uses the RDS APIs and does not depend on cloudformation support.

And terraform seems to support it as well. https://registry.terraform.io/modules/terraform-aws-modules/rds-aurora/aws/latest/examples/serverless. Not the first time I've seen terraform having better support for AWS resources than cloudformation.

[demiurg]

Guys, this would be great, currently have to make this from console instead of IaC.

[ccggeo]

Also bumping. This should definitely be native to CFN

[vincent-dm]

We are seriously blocked by this. Please prioritize it, thanks!

[shaheerkhan12605]

The closest thing now seem to be

1. Deploy AWS::RDS::DBCluster with EngineMode: provision
2. Run aws rds modify-db-cluster --serverless-v2-scaling-configuration '{"MinCapacity":0.5,"MaxCapacity":128}' --db-cluster-identifier <ClusterId>, if you have not find this config, update the awscli
3. Deploy AWS::RDS::DBInstance with DBInstanceClass: db.serverless

@kftsehk I am using the following shell script to create the serverless v2

CLUSTER_NAME=$(cat rds_serverless.dev.json | jq -r .ClusterName)
NumberOfInstances=$(cat rds_serverless.dev.json | jq -r .NumberOfInstances) # 1 or 2
ENVIRONMENT=$(cat rds_serverless.dev.json | jq -r .Environment)
MinCapacity=$(cat rds_serverless.dev.json | jq -r .MinCapacity)
MaxCapacity=$(cat rds_serverless.dev.json | jq -r .MaxCapacity)

expr $NumberOfInstances + 0 # Converting to numeric

function tryexec() {
    "$@"
    retval=$?
    [[ $retval -eq 0 ]] && return 0
    echo "A command has failed:"
    echo "  $*"
    echo "Value returned: ${retval}"
   exit 254
}
tryexec aws rds --region $REGION modify-db-cluster --db-cluster-identifier $CLUSTER_NAME-cluster --serverless-v2-scaling-configuration MinCapacity=$MinCapacity,MaxCapacity=$MaxCapacity --output text
tryexec aws rds --region $REGION create-db-instance --db-instance-identifier $CLUSTER_NAME-primary-instance --db-cluster-identifier $CLUSTER_NAME-cluster --db-instance-class db.serverless --engine $engine --output text
tryexec aws rds --region $REGION wait db-instance-available --db-instance-identifier $CLUSTER_NAME-primary-instance --output text
if [[ $NumberOfInstances -gt 1 ]]; then
    tryexec aws rds --region $REGION create-db-instance --db-instance-identifier $CLUSTER_NAME-primary-instance-2 --db-cluster-identifier $CLUSTER_NAME-cluster --db-instance-class db.serverless --engine $engine --output text
    tryexec aws rds --region $REGION wait db-instance-available --db-instance-identifier $CLUSTER_NAME-primary-instance-2
fi
tryexec aws rds --region $REGION failover-db-cluster --db-cluster-identifier $CLUSTER_NAME-cluster --target-db-instance-identifier $CLUSTER_NAME-primary-instance --output text
tryexec aws rds --region $REGION delete-db-instance --db-instance-identifier $CLUSTER_NAME-primary --skip-final-snapshot --output text
tryexec aws rds --region $REGION wait db-instance-deleted --db-instance-identifier $CLUSTER_NAME-primary --output text

[matart15]

It is 2nd most 👍 issue. If we combine #1153 the it is the most 👍 issue.

not to mention CDK repo have same request with the most 👍 .

But it is not on public milestone

[mgirard772]

+1

[RajasGujarathi]

The closest thing now seem to be

1. Deploy `AWS::RDS::DBCluster` with `EngineMode: provision`

2. Run `aws rds modify-db-cluster --serverless-v2-scaling-configuration '{"MinCapacity":0.5,"MaxCapacity":128}' --db-cluster-identifier <ClusterId>`, if you have not find this config, update the `awscli`

3. Deploy `AWS::RDS::DBInstance` with `DBInstanceClass: db.serverless`

Thanks, this worked smoothly 🙏

[Beast12]

For Anyone that is trying to put this all in cloudformation...

CommandRunnerServerlessV2:
    DependsOn: RDSClusterServerlessV2
    Condition: IsServerless-v2
    Type: AWSUtility::CloudFormation::CommandRunner
    Properties:
      Command: !Sub |
        sudo rm -rf /usr/local/aws && curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" && unzip awscliv2.zip && sudo ./aws/install && \
        aws rds modify-db-cluster --serverless-v2-scaling-configuration '{"MinCapacity":0.5,"MaxCapacity":8}' --db-cluster-identifier ${RDSClusterServerlessV2} --region ${AWS::Region} \
        > /command-output.txt
      LogGroup: CloudRunner
      SubnetId:
        !Select [0, !FindInMap [Environments, !Ref Environment, PrivateSubnets]]
      Role: !Ref CommandRunnerRole
      InstanceType: t2.micro

Make sure the Role (Which refers to the instance profile btw...) has enough permissions to run the command of course

[ekeyser]

any ETA on this one?

@driverpt

Just heard back from AWS Support, and they said

Having said that, as you know any new feature or a fix requires the product to undergo rigorous testing and troubleshooting before it is made available, to ensure security and stability for end users. Unfortunately, I will not be able to provide you an ETA for it at the moment as the internal team does not share this information with us.

However, you also can track the Aurora Serverless v2 coverage via the CloudFormation GitHub Roadmap. There is currently an opened issue discussing this use case here [2][3].

The same standard reply from support re no ETA. How surprising.

BTW, this response about rigorous testing is complete garbage but I'm sure plenty of people will just lap it up as a valid excuse. They need to take responsibility and admit that releasing a service or product involves coordination with other teams and testing beforehand. This is what it means to be a professional company and a professional software development organization and their rhetoric surrounding this is dogsh*t. What they're doing instead is releasing products in the market in hopes of obtaining a market advantage but which are not integrated with the rest of the AWS service offerings and tooling. This in turn just makes everyone's lives supporting these half-baked offerings miserable.

This is the hallmark of a company that is no longer a gold standard and which is simply just churning out garbage.

[relsunkaev]

@ekeyser

There is actually a soft ETA, as per this reply.

[skysb]

Seems like this has been implemented already right?
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-rds-dbcluster.html

At the section
Creating an Amazon Aurora Serverless v2 DB cluster
??
or am I mistaken?

[zsims]

You're right @skysb, seems it was implemented and this issue hasn't been updated.

I tried it out and can confirm it works well 👍

[moltar]

woohoo party!

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-rds-dbcluster.html#cfn-rds-dbcluster-serverlessv2scalingconfiguration

[osdrv]

Hey all, the feature is now available globally. Please refer to the doc for more details. Please open a new ticket upon any other issue concerning ServerlessV2. This ticket might be closed now.

[trygve-aaberge-adsk]

@osdrv: Great, thanks! However I'm not finding any documentation on how to create a DBInstance under a ServerlessV2 cluster, and if I try I get Invalid DB Instance class: null. I'm not sure what I would set the DB instance class to, do you know?

[Beast12]

you should choose the db.serverless

[trygve-aaberge-adsk]

@Beast12: Ah, thanks!

[mmuller88]

Great work but looks like it first having its first bug with not serving Postgres serverless v2 aws/aws-cdk#20197 (comment)

[Beast12]

It does work in eu-west-1. I've created 4 aurora-postgres clusters with cfn. Version 14.3

[trygve-aaberge-adsk]

@mmuller88: I see the error in that issue is The engine mode serverless you requested is currently unavailable. I think I got that error when trying to set EngineMode: serverless. It worked after removing that. Make sure you (or CDK) is not setting this parameter when using Serverless v2.

[mmuller88]

@trygve-aaberge-adsk wow that fixed it. Thanks :)

[mmuller88]

Sorry me again. Even if the deploy went smoothly it doesn't create the writer instance. So it seems there is still something wrong or I miss something ^^.

[trygve-aaberge-adsk]

@mmuller88: You need to create one or more database instances in the cluster in addition to the cluster itself. Like this (this is CloudFormation YAML, I don't use CDK so don't know how to do it there):

    DatabaseInstance1:
      Type: AWS::RDS::DBInstance
      Properties:
        DBClusterIdentifier: !Ref DatabaseCluster
        Engine: aurora-postgresql
        DBInstanceClass: db.serverless

[mmuller88]

@trygve-aaberge-adsk got it thanks :)

[aadityapmr26]

Request you if you could share the Yaml file for deploying server less V2 which is working for you

[serge-salamanka]

the documentation example is missing a guidence to create an instance. it is not obvious for newcommers

[Beast12]

Sorry... Didn't had the time yet... This is a complete example... It has some basics from a aws aurora cluster cloudformation. But I added some stuff for our pupose... So you do need to filter maybe... But at least it has the instances...

AWSTemplateFormatVersion: 2010-09-09
Description: Template Containing Serverless V2 Aurora
Parameters:
  AcuMin:
    Default: 0.5
    Description: Minimum ACU Capacity
    Type: String
  AcuMax:
    Default: 8
    Description: Maximum ACU Capacity
    Type: String
  ApplicationName:
    Default: ToDo
    Description: Applications can consist of multiple resources. The application should group all resources that conceptually belong to the same application. (Or service offering)
    Type: String
  Availability:
    AllowedValues:
      - 8/5
      - 8/7
      - 24/5
      - 24/7
      - 12/5
      - 12/7
    Default: 24/7
    Description: hoursPerDay/daysPerWeek
    Type: String
  ContactPerson:
    Default: [email protected]
    Description: ContactPerson for the Application
    Type: String
  DBEngine:
    AllowedValues:
      - aurora
      - aurora-mysql
      - aurora-postgresql
    Default: aurora-postgresql
    Description: The name of the database engine that you want to use for this DB instance.
    Type: String
  DBEngineVersion:
    AllowedValues:
      - 14.3
      - 13.7
      - 10.18.3
    Default: 14.3
    Description: The version number of the database engine to use.
    Type: String
  DBInstanceClass:
    Default: db.t4g.medium
    Description: Database Instance Class
    Type: String
    AllowedValues:
      - db.serverless
      - db.t4g.medium
      - db.t4g.large
      - db.r5.large
      - db.r5.xlarge
      - db.r5.2xlarge
      - db.r5.4xlarge
      - db.r5.8xlarge
      - db.r5.12xlarge
      - db.r5.16xlarge
      - db.r5.24xlarge
      - db.r6g.large
      - db.r6g.xlarge
      - db.r6g.2xlarge
      - db.r6g.4xlarge
      - db.r6g.8xlarge
      - db.r6g.12xlarge
      - db.r6g.16xlarge
      - db.x2g.large
      - db.x2g.xlarge
      - db.x2g.2xlarge
      - db.x2g.4xlarge
      - db.x2g.8xlarge
      - db.x2g.12xlarge
      - db.x2g.16xlarge
  DBPrivateDNSRecord:
    Default: db.venly-staging.local
    Description: Internal DNS record to create
    Type: String
  DBSecurityGroup:
    Description: Select the Security Group to use for the ECS cluster rds
    Type: AWS::EC2::SecurityGroup::Id
  DBSnapshotIdentifier:
    Default: ""
    Description: The RDS MariaDB snapshot name to restore to the new DB instance
    Type: String
  DBUser:
    Default: admin
    Description: The username to use for the RDS instance
    Type: String
  DataBaseName:
    ConstraintDescription: Must contain 1 to 63 letters
    Default: todo
    Description: Name of the Database to create
    Type: String
  EnableIAM:
    AllowedValues:
      - true
      - false
    ConstraintDescription: must be true or false.
    Default: false
    Description: A value that indicates whether to enable mapping of AWS Identity and Access Management (IAM) accounts to database accounts.
    Type: String
  Environment:
    AllowedValues:
      - qa
      - staging
      - prd
      - nonprd
      - resources
      - dev
      - prd-sbx
    Description: Choose between known environments dev, uat, prd, dev, prd-sbx
    Type: String
  ProjectName:
    Default: ToDo
    Description: Name of the project this resource is part of
    Type: String
Mappings:
  DBFamilyMap:
    "11.16":
      "family": "aurora-postgresql11"
    "12.11":
      "family": "aurora-postgresql12"
    "13.7":
      "family": "aurora-postgresql13"
    "14.3":
      "family": "aurora-postgresql14"
  Fn::Transform:
    Name: AWS::Include
    Parameters:
      Location: s3://macros/Mappings/venly-aws-mappings.yml
Conditions:
  # Check if Snapshot should be used to restore DB
  UseDbSnapshot:
    Fn::Not:
      - Fn::Equals:
          - Ref: DBSnapshotIdentifier
          - ""
  IsPrdOrStaging:
    !Or [
      !Equals [!Ref Environment, prd],
      !Equals [!Ref Environment, staging],
      !Equals [!Ref Environment, prd-sbx],
    ]
  IsPrd: !Equals [!Ref Environment, prd]
  IsPostgresql: !Equals [!Ref DBEngine, aurora-postgresql]
  IsServerless: !Equals [!Ref DBInstanceClass, db.serverless]
Resources:
  EnhancedMonitoringRole:
    Condition: IsPrd
    Type: "AWS::IAM::Role"
    Properties:
      ManagedPolicyArns:
        - "arn:aws:iam::aws:policy/service-role/AmazonRDSEnhancedMonitoringRole"
      AssumeRolePolicyDocument:
        Version: "2008-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: "monitoring.rds.amazonaws.com"
            Action: "sts:AssumeRole"
  RDSDBClusterParameterGroup:
    Type: AWS::RDS::DBClusterParameterGroup
    Properties:
      Description:
        !Join [
          "- ",
          [
            "Aurora PG Cluster Parameter Group for  Cloudformation Stack ",
            !Ref DataBaseName,
          ],
        ]
      Family: !FindInMap [DBFamilyMap, !Ref DBEngineVersion, "family"]
      Parameters:
        rds.force_ssl: 1
  DBParamGroup:
    Type: AWS::RDS::DBParameterGroup
    Properties:
      Description:
        !Join [
          "- ",
          [
            "Aurora PG Database Instance Parameter Group for Cloudformation Stack ",
            !Ref DataBaseName,
          ],
        ]
      Family: !FindInMap [DBFamilyMap, !Ref DBEngineVersion, "family"]
      Parameters:
        shared_preload_libraries: auto_explain,pg_stat_statements,pg_hint_plan,pgaudit
        log_statement: "ddl"
        log_connections: 1
        log_disconnections: 1
        log_lock_waits: 1
        log_min_duration_statement: 5000
        auto_explain.log_min_duration: 5000
        auto_explain.log_verbose: 1
        log_rotation_age: 1440
        log_rotation_size: 102400
        rds.log_retention_period: 10080
        random_page_cost: 1
        track_activity_query_size: 16384
        idle_in_transaction_session_timeout: 7200000
        statement_timeout: 7200000
        search_path: '"$user",public'
  AuroraKMSCMK:
    Condition: IsPrd
    Type: "AWS::KMS::Key"
    Properties:
      KeyPolicy:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              AWS: !Sub "arn:aws:iam::${AWS::AccountId}:root"
            Action: "kms:*"
            Resource: "*"
          - Effect: Allow
            Principal:
              AWS: "*"
            Action:
              - "kms:Encrypt"
              - "kms:Decrypt"
              - "kms:ReEncrypt*"
              - "kms:GenerateDataKey*"
              - "kms:CreateGrant"
              - "kms:ListGrants"
              - "kms:DescribeKey"
            Resource: "*"
            Condition:
              StringEquals:
                "kms:CallerAccount": !Ref "AWS::AccountId"
                "kms:ViaService": !Sub "rds.${AWS::Region}.amazonaws.com"
  AuroraKMSCMKAlias:
    Condition: IsPrd
    Type: "AWS::KMS::Alias"
    DependsOn: AuroraDBCluster
    Properties:
      AliasName: !Sub "alias/${AuroraDBCluster}"
      TargetKeyId: !Ref AuroraKMSCMK
  RdsSecret:
    Type: AWS::SecretsManager::Secret
    Properties:
      Description: !Sub ${ApplicationName}-${Environment}-rds-secret
      Name: !Sub ${ApplicationName}-${Environment}-rds-secret
      GenerateSecretString:
        SecretStringTemplate: !Sub '{"username": "${DBUser}"}'
        GenerateStringKey: password
        ExcludePunctuation: true
        PasswordLength: 41
      Tags:
        - Key: ApplicationName
          Value: !Ref ApplicationName
        - Key: Availability
          Value: !Ref Availability
        - Key: ContactPerson
          Value: !Ref ContactPerson
        - Key: Environment
          Value: !Ref Environment
        - Key: Monitor
          Value: !If
            - IsPrdOrStaging
            - true
            - false
        - Key: ProjectName
          Value: !Ref ProjectName
        - Key: Name
          Value: !Sub ${ApplicationName}${Environment}-RDS-Secret
  SecretTargetAttachment:
    Type: AWS::SecretsManager::SecretTargetAttachment
    Properties:
      TargetId: !Ref AuroraDBCluster
      SecretId: !Ref RdsSecret
      TargetType: AWS::RDS::DBCluster
  RdsClusterSubnetGroup:
    Type: AWS::RDS::DBSubnetGroup
    Properties:
      DBSubnetGroupDescription: !Sub ${DBEngine}-${ApplicationName}-${Environment}-DBSubnetGroup
      DBSubnetGroupName: !Sub ${DBEngine}-${ApplicationName}-${Environment}-DBSubnetGroup
      SubnetIds:
        - !Select [
            0,
            !FindInMap [Environments, !Ref Environment, PrivateSubnets],
          ]
        - !Select [
            1,
            !FindInMap [Environments, !Ref Environment, PrivateSubnets],
          ]
        - !Select [
            2,
            !FindInMap [Environments, !Ref Environment, PrivateSubnets],
          ]
      Tags:
        - Key: ApplicationName
          Value: !Ref ApplicationName
        - Key: Availability
          Value: !Ref Availability
        - Key: ContactPerson
          Value: !Ref ContactPerson
        - Key: Environment
          Value: !Ref Environment
        - Key: Monitor
          Value: !If
            - IsPrdOrStaging
            - true
            - false
        - Key: ProjectName
          Value: !Ref ProjectName
  AuroraDBCluster:
    Type: AWS::RDS::DBCluster
    DeletionPolicy: Snapshot
    UpdateReplacePolicy: Snapshot
    Properties:
      BackupRetentionPeriod: !If
        - IsPrdOrStaging
        - 5
        - 2
      DBClusterIdentifier: !Sub ${ApplicationName}-${Environment}-RDS
      DBSubnetGroupName: !Ref RdsClusterSubnetGroup
      DatabaseName:
        Fn::If:
          - UseDbSnapshot
          - Ref: AWS::NoValue
          - Ref: DataBaseName
      DeletionProtection: true
      # EnableCloudwatchLogsExports:
      #   - postgresql
      EnableIAMDatabaseAuthentication: !Ref EnableIAM
      Engine: !Ref DBEngine
      EngineVersion: !Ref DBEngineVersion
      KmsKeyId: !If
        - UseDbSnapshot
        - !Ref AWS::NoValue
        - !If
          - IsPrd
          - !Ref AuroraKMSCMK
          - !Ref AWS::NoValue
      MasterUserPassword:
        Fn::If:
          - UseDbSnapshot
          - Ref: AWS::NoValue
          - !Sub "{{resolve:secretsmanager:${RdsSecret}::password}}"
      MasterUsername:
        Fn::If:
          - UseDbSnapshot
          - Ref: AWS::NoValue
          - !Sub "{{resolve:secretsmanager:${RdsSecret}::username}}"
      Port: !If
        - IsPostgresql
        - 5432
        - 3306
      ServerlessV2ScalingConfiguration: !If
        - IsServerless
        - MinCapacity: !Ref AcuMin
          MaxCapacity: !Ref AcuMax
        - !Ref AWS::NoValue
      SnapshotIdentifier:
        Fn::If:
          - UseDbSnapshot
          - Ref: DBSnapshotIdentifier
          - Ref: AWS::NoValue
      StorageEncrypted: !If [UseDbSnapshot, !Ref "AWS::NoValue", true]
      VpcSecurityGroupIds:
        - !Ref DBSecurityGroup
      Tags:
        - Key: ApplicationName
          Value: !Ref ApplicationName
        - Key: Availability
          Value: !Ref Availability
        - Key: ContactPerson
          Value: !Ref ContactPerson
        - Key: Environment
          Value: !Ref Environment
        - Key: Monitor
          Value: !If
            - IsPrdOrStaging
            - true
            - false
        - Key: ProjectName
          Value: !Ref ProjectName
        - Key: Name
          Value: !Sub ${ProjectName}-${Environment}-rds
  AuroraDBFirstInstance:
    Type: "AWS::RDS::DBInstance"
    Properties:
      AllowMajorVersionUpgrade: false
      AutoMinorVersionUpgrade: !If
        - IsPrd
        - false
        - true
      CopyTagsToSnapshot: true
      DBClusterIdentifier: !Ref AuroraDBCluster
      DBInstanceClass: !Ref DBInstanceClass
      DBParameterGroupName: !Ref DBParamGroup
      DBSubnetGroupName: !Ref RdsClusterSubnetGroup
      EnablePerformanceInsights: true
      PerformanceInsightsKMSKeyId: !If
        - IsPrd
        - !Ref AuroraKMSCMK
        - !Ref AWS::NoValue
      Engine: !Ref DBEngine
      MonitoringInterval: !If
        - IsPrd
        - 60
        - 0
      MonitoringRoleArn: !If
        - IsPrd
        - !GetAtt EnhancedMonitoringRole.Arn
        - !Ref AWS::NoValue
      Tags:
        - Key: ApplicationName
          Value: !Ref ApplicationName
        - Key: Availability
          Value: !Ref Availability
        - Key: ContactPerson
          Value: !Ref ContactPerson
        - Key: Environment
          Value: !Ref Environment
        - Key: Monitor
          Value: !If
            - IsPrdOrStaging
            - true
            - false
        - Key: ProjectName
          Value: !Ref ProjectName
        - Key: Name
          Value: !Sub ${ProjectName}-${Environment}-rds
  AuroraDBSecondInstance:
    Type: "AWS::RDS::DBInstance"
    Condition: IsPrd
    DependsOn:
      - AuroraDBFirstInstance
    Properties:
      AllowMajorVersionUpgrade: false
      AutoMinorVersionUpgrade: !If
        - IsPrd
        - false
        - true
      CopyTagsToSnapshot: true
      DBClusterIdentifier: !Ref AuroraDBCluster
      DBInstanceClass: !Ref DBInstanceClass
      DBParameterGroupName: !Ref DBParamGroup
      DBSubnetGroupName: !Ref RdsClusterSubnetGroup
      EnablePerformanceInsights: true
      PerformanceInsightsKMSKeyId: !Ref AuroraKMSCMK
      Engine: !Ref DBEngine
      MonitoringInterval: !If
        - IsPrd
        - 60
        - 0
      MonitoringRoleArn: !If
        - IsPrd
        - !GetAtt EnhancedMonitoringRole.Arn
        - !Ref AWS::NoValue
      Tags:
        - Key: ApplicationName
          Value: !Ref ApplicationName
        - Key: Availability
          Value: !Ref Availability
        - Key: ContactPerson
          Value: !Ref ContactPerson
        - Key: Environment
          Value: !Ref Environment
        - Key: Monitor
          Value: !If
            - IsPrdOrStaging
            - true
            - false
        - Key: ProjectName
          Value: !Ref ProjectName
        - Key: Name
          Value: !Sub ${ProjectName}-${Environment}-rds
  RdsPrivateDNSRecord:
    Type: AWS::Route53::RecordSetGroup
    Properties:
      HostedZoneId: !FindInMap [Environments, !Ref Environment, HostedZoneID]
      RecordSets:
        - Name: !Ref DBPrivateDNSRecord
          Type: CNAME
          TTL: 60
          ResourceRecords:
            - !GetAtt AuroraDBCluster.Endpoint.Address
Outputs:
  ClusterEndpoint:
    Description: "Aurora Cluster/Writer Endpoint"
    Value: !GetAtt "AuroraDBCluster.Endpoint.Address"
  ReaderEndpoint:
    Description: "Aurora Reader Endpoint"
    Value: !GetAtt "AuroraDBCluster.ReadEndpoint.Address"
  RdsSecret:
    Description: Returns the ARN of the secret configured
    Value: !Ref RdsSecret
  Port:
    Description: "Aurora Endpoint Port"
    Value: !GetAtt "AuroraDBCluster.Endpoint.Port"
  DBUsername:
    Description: "Database master username"
    Value: !Ref DBUser
  DBName:
    Description: "Database Name"
    Value: !Ref DataBaseName
  PSQLCommandLine:
    Description: PSQL Command Line
    Value: !Join
      - ""
      - - "psql --host="
        - !GetAtt "AuroraDBCluster.Endpoint.Address"
        - " --port="
        - !GetAtt "AuroraDBCluster.Endpoint.Port"
        - " --username="
        - !Ref DBUser
        - " --dbname="
        - !Ref DataBaseName
  MasterJDBCConnectionString:
    Description: JDBC connection string for the master database
    Value:
      !Join [
        "",
        [
          "jdbc:postgresql://",
          !GetAtt [AuroraDBCluster, Endpoint.Address],
          ":",
          !GetAtt [AuroraDBCluster, Endpoint.Port],
          /,
          !Ref "DataBaseName",
        ],
      ]
  MasterConnectionString:
    Description: JDBC connection string for the master database
    Value:
      !Join [
        "",
        [
          !GetAtt [AuroraDBCluster, Endpoint.Address],
          ":",
          !GetAtt [AuroraDBCluster, Endpoint.Port],
          /,
          !Ref "DataBaseName",
        ],
      ]

[skysb]

As a side note, when I actually tried it, I was forced to put in the engine version as "8.0.mysql_aurora.3.02.0" and create the DB instance, contradicting the documentation that with only the cluster, the database would be ready to use.
Without the engine version in the db cluster, it kept creating the mysql version 5.7 for me

[siebrand]

Here is a tested and working stripped down example for an RDS Aurora for MySQL 5.7 cluster with a single instance that is updated to MySQL 8.0 and RDS Serverless V2. See comments in CloudFormation for the step to follow.

---
AWSTemplateFormatVersion: 2010-09-09
Description: Example stack for migration from RDS Aurora MySQL 5.7 to Aurora Serverless MySQL 8.0
#
#   Migration from RDS Aurora MySQL 5.7 to Aurora Serverless MySQL 8.0 in multiple steps
#
#   1. Set RdsParameterGroupToUse to "default"
#   2. Set RdsParameterGroupToUse to "new" and RdsVersion to "80"
#   3. Once completed, the legacy code can be removed from the template.
#
Parameters:
  RdsParameterGroupToUse:
    Type: String
    Default: old
    AllowedValues:
      - old
      - default
      - new
    Description: Which RDS parameter group to use. Must update to default before updating RdsVersion MySQL 5.7 to 8.0 serverless.
  RdsVersion:
    Type: String
    Default: 57
    AllowedValues:
      - 57
      - 80
    Description: Which RDS MySQL version to use.
Conditions:
  UseParameterGroupOld: !Equals [ !Ref RdsParameterGroupToUse, 'old' ]
  UseParameterGroupDefault: !Equals [ !Ref RdsParameterGroupToUse, 'default' ]
  UseMySQL57: !Equals [ !Ref RdsVersion, '57' ]
  UseMySQL80: !Equals [ !Ref RdsVersion, '80' ]
Resources:
  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 192.168.0.0/24
  SubnetPrivate1:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone: !Sub ${AWS::Region}a
      CidrBlock: 192.168.0.64/26
      VpcId: !Ref VPC
  SubnetPrivate2:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone: !Sub ${AWS::Region}b
      CidrBlock: 192.168.0.128/26
      VpcId: !Ref VPC
  SgAllowAccessToMySQL:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: !Sub Security group for ${AWS::StackName}
      VpcId: !Ref VPC
  SgRdsAurora:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: !Sub Security group for ${AWS::StackName}
      VpcId: !Ref VPC
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 3306
          ToPort: 3306
          SourceSecurityGroupId: !Ref SgAllowAccessToMySQL
          Description: MySQL/3306 from SgAllowAccessToMySQL
  RdsAuroraSubnetGroup:
    Type: AWS::RDS::DBSubnetGroup
    Properties:
      DBSubnetGroupDescription: Subnet group for Aurora database
      SubnetIds:
        - !Ref SubnetPrivate1
        - !Ref SubnetPrivate2
  RDSDBClusterParameterGroup:
    Type: AWS::RDS::DBClusterParameterGroup
    Properties:
      Description: !Sub Database cluster parameter group for RDS Aurora ${AWS::StackName}
      Family: aurora-mysql5.7
      Parameters:
        tls_version: TLSv1.2
  RDSDBClusterParameterGroupMySQL8:
    Type: AWS::RDS::DBClusterParameterGroup
    Properties:
      Description: !Sub Database cluster parameter group for RDS Aurora ${AWS::StackName}
      Family: aurora-mysql8.0
      Parameters:
        tls_version: TLSv1.2
  RdsAuroraParameterGroup:
    Type: AWS::RDS::DBParameterGroup
    Properties:
      Description: !Sub Database parameter group for RDS Aurora ${AWS::StackName}
      Family: aurora-mysql5.7
      Parameters:
        performance_schema: 0
  RdsAuroraParameterGroupMySQL8:
    Type: AWS::RDS::DBParameterGroup
    Properties:
      Description: !Sub Database parameter group for RDS Aurora ${AWS::StackName}
      Family: aurora-mysql8.0
      Parameters:
        performance_schema: 0
  RdsAuroraCluster:
    Type: AWS::RDS::DBCluster
    Properties:
      ServerlessV2ScalingConfiguration: !If
        - UseMySQL80
        - MinCapacity: 0.5
          MaxCapacity: 8
        - !Ref AWS::NoValue
      StorageEncrypted: true
      DBSubnetGroupName: !Ref RdsAuroraSubnetGroup
      DBClusterParameterGroupName:  !If [ UseMySQL57, !Ref RDSDBClusterParameterGroup, !Ref RDSDBClusterParameterGroupMySQL8 ]
      Engine: aurora-mysql
      # aws rds describe-orderable-db-instance-options --engine aurora-mysql --db-instance-class db.serverless --query 'OrderableDBInstanceOptions[].[EngineVersion]' --output text --region my_region
      # Options currently are 8.0.mysql_aurora.3.02.0 and 8.0.mysql_aurora.3.02.1
      EngineVersion: !If [ UseMySQL57, '5.7.mysql_aurora.2.10.2', '8.0.mysql_aurora.3.02.1' ]
      MasterUsername: root
      MasterUserPassword: justapassword
      Port: '3306'
      VpcSecurityGroupIds:
        - !Ref SgRdsAurora
  AuroraPrimaryInstance:
    Type: AWS::RDS::DBInstance
    Properties:
      DBClusterIdentifier: !Ref RdsAuroraCluster
      DBInstanceClass: !If [ UseMySQL57, 'db.t4g.medium', 'db.serverless' ]
      DBSubnetGroupName: !Ref RdsAuroraSubnetGroup
      DBParameterGroupName: !If [ UseParameterGroupOld, !Ref RdsAuroraParameterGroup, !If [ UseParameterGroupDefault, 'default.aurora-mysql5.7', !Ref RdsAuroraParameterGroupMySQL8 ] ]
      Engine: aurora-mysql
      EnablePerformanceInsights: false

[jankammerath]

@siebrand thanks a lot, your sample really helped me get started. Do you have an idea on how to do Multi-AZ?

I tried setting the AvailabilityZone property for each DBInstance, but got rejected with:
Properties validation failed for resource DatabaseInstanceAZ2 with message: #: extraneous key [AvailaibilityZone] is not permitted

When I create two AWS::RDS::DBInstance in my CloudFormation template, it deploys both in the same AZ 🫠

[trygve-aaberge-adsk]

@jankammerath: You should be able to set AvailabilityZone on the instance. Make sure to set it under Properties. Here is an example with a multi-AZ cluster with tree instances in different zones:

DatabaseCluster:
  Type: AWS::RDS::DBCluster
  Properties:
    DBClusterIdentifier: ...
    DatabaseName: ...
    Engine: aurora-postgresql
    EngineVersion: 14.4
    MasterUsername: ...
    MasterUserPassword: ...
    Port: 5432
    DBSubnetGroupName: !Ref DatabaseSubnetGroup
    VpcSecurityGroupIds:
      - !Ref DatabaseSecurityGroup
    ServerlessV2ScalingConfiguration:
      MinCapacity: 1
      MaxCapacity: 64

DatabaseInstance1:
  Type: AWS::RDS::DBInstance
  Properties:
    DBClusterIdentifier: !Ref DatabaseCluster
    Engine: aurora-postgresql
    DBInstanceClass: db.serverless
    AvailabilityZone: eu-west-1a

DatabaseInstance2:
  Type: AWS::RDS::DBInstance
  Properties:
    DBClusterIdentifier: !Ref DatabaseCluster
    Engine: aurora-postgresql
    DBInstanceClass: db.serverless
    AvailabilityZone: eu-west-1b

DatabaseInstance3:
  Type: AWS::RDS::DBInstance
  Properties:
    DBClusterIdentifier: !Ref DatabaseCluster
    Engine: aurora-postgresql
    DBInstanceClass: db.serverless
    AvailabilityZone: eu-west-1c

DatabaseSubnetGroup:
  Type: AWS::RDS::DBSubnetGroup
  Properties:
    DBSubnetGroupDescription: ...
    SubnetIds:
      - ...
      - ...

DatabaseSecurityGroup:
  Type: AWS::EC2::SecurityGroup
  Properties:
    GroupDescription: ...
    VpcId: ...

[jankammerath]

@trygve-aaberge-adsk Awesome, thank you. It's working now. I probably wasn't careful enough placing the AvailabilityZone key properly under the properties.