eks: ALB Controller fails to create due to lack of AddTag permissions

[juweeks]

Describe the bug

Trying to launch an EKS cluster (v1.23) with an ALB controller (v2.4.1). The cluster launches successfully, but the ALB does not. A cluster with nearly the exact same configuration was launched less than a week ago (7/14/23) successfully in another account with an ALB. No error is being returned by the resource that creates the ALB. It just doesn’t get created.

Looking at the Cloudtrail logs, I noticed that the role for the ALB controller gets an AccessDenied when running a CreateLoadBalancer with “not authorized to perform: elasticloadbalancing:AddTags”. Somewhat odd, since that permission is on the role.

Expected Behavior

EKS launches with ALB Controller

Current Behavior

EKS launches successfully but ALB does not and no accompanying CloudFormation error.

Cloudtrail error:

AccessDenied 
User: arn:aws:sts::xxx:assumed-role/eksbclusteralbcontroller-9999/123456 is not authorized to perform: elasticloadbalancing:AddTags on resource: arn:aws:elasticloadbalancing:us-east-1:xxx:loadbalancer/app/cluster-albingre-5555/* because no identity-based policy allows the elasticloadbalancing:AddTags action 

Reproduction Steps

from aws_cdk import aws_eks as eks
alb_controller = eks.AlbController(
                self,
                id="alb_controller",
                cluster=self.cluster,
                version=eks.AlbControllerVersion.V2_4_1,
            )

Possible Solution

deploys successfully when the permission is added via the node:

cfn_alb_role = alb_controller.node.try_find_child("alb-sa").node.try_find_child("Role")
arn_resource = Stack.of(self).format_arn(
                service="elasticloadbalancing", 
                resource="loadbalancer/app", 
                resource_name="*/*"
            )
cfn_alb_role.add_to_policy(
    iam.PolicyStatement(
        actions=["elasticloadbalancing:AddTags"],
        resources=[arn_resource],
    )
)

Additional Information/Context

This problem seems to have occurred between 7/14/23 (last successful EKS launch w/ALB) and 7/18/23 (first failed EKS launch a/ALB)

CDK CLI Version

2.78

Framework Version

No response

Node.js Version

16.15.1

OS

Mac 13.4.1 (Ventura)

Language

Python

Language Version

3.9.6

Other information

No response

[pahud]

According to this:

aws-cdk/packages/aws-cdk-lib/aws-eks/lib/alb-controller.ts

Lines 262 to 267 in 3e47d1b

	// https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/deploy/installation/#iam-permissions
	const policy: any = props.policy ?? JSON.parse(fs.readFileSync(path.join(__dirname, 'addons', `alb-iam_policy-${props.version.version}.json`), 'utf8'));
	for (const statement of policy.Statement) {
	serviceAccount.addToPrincipalPolicy(iam.PolicyStatement.fromJson(statement));
	}

The required permissions are defined in
https:/aws/aws-cdk/blob/main/packages/aws-cdk-lib/aws-eks/lib/addons/alb-iam_policy-v2.4.1.json

But this should have sastified the required permissions?

aws-cdk/packages/aws-cdk-lib/aws-eks/lib/addons/alb-iam_policy-v2.4.1.json

Lines 150 to 158 in 3e47d1b

	"Effect": "Allow",
	"Action": [
	"elasticloadbalancing:AddTags",
	"elasticloadbalancing:RemoveTags"
	],
	"Resource": [
	"arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
	"arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
	"arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"

[juweeks]

But this should have sastified the required permissions?

exactly. that's what I don't understand. the only difference is when the Condition is left out, it deploys fine. Even weirder that it fails silently- Cloudformation doesn't raise anything.

Idk. maybe i'm crazy but this same configuration and code worked last week. maybe an issue with the custom resource? Cython came out with a release during this time that's broken things- namely PyYaml- and noticed that it's been cited in several downstream issues with AWS resources. of course that might not apply here since that's mostly JS.

don't want to lead you down a long path. i've got a fix that works. so not too high of a priority on my end. if you notice there are some more issues coming in about it or whenever you launch a new cluster w/ALB.

[juweeks]

this issue appears to be fixed with ALB controller v2.4.7

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.