(aws-eks): Exposing the Auto Scaling Group from a Nodegroup

[gjmveloso]

Description

The idea of this feature request is to add to EKS Construct Library the ability of exposing the resulting Auto Scaling Group created for a EKS Managed Node Group

Use Case

Enable Interacting with the resulting Auto Scaling Group created by addNodegroupCapacity method

Proposed Solution

N/A

Other information

No response

Acknowledge

• I may be able to implement this feature request
• This feature might incur a breaking change

[otaviomacedo]

Hi, @gjmveloso. What kind of interaction do you expect to do? What is your use case?

FYI, addNodegroupCapacity does return the Nodegroup that it creates.

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[David-Tamrazov]

This would help me as well. In my usecase, I want to be able to set up session manager for the managed nodegroup I created. This requires me to add permissions to the autoscaling group role and allow connections to and from it on some ports. In a non-managed nodegroup context, I would do it like so:

autoScalingGroup.role.addToPrincipalPolicy(new PolicyStatement({
    effect: iam.Effect.ALLOW,
    actions: [
        'ssm:UpdateInstanceInformation',
        'ssmmessages:CreateControlChannel',
        'ssmmessages:CreateDataChannel',
        'ssmmessages:OpenControlChannel',
        'ssmmessages:OpenDataChannel',
    ],
    resources: [
        '*',
    ],
}));
autoScalingGroup.connections.allowFromAnyIpv4(
    ec2.Port.tcp(22),
    'inbound traffic for SSH into the instance',
);
autoScalingGroup.connections.allowToAnyIpv4(
    ec2.Port.tcp(443),
    'outbound traffic rule to port 443 for AWS Session Manager',
);

However even though addNodegroupCapacity returns the created Nodegroup, there is no way to access the underlying Autoscaling group. A property on the aws_eks.Nodegroup construct to access the autoscaling group would unblock this approach.

[yws-ss]

Hello team,

There have a scenario need this feature. Per Cluster Autoscaler document mentioned, if the node group want to scale up from 0, we need to add the resources tags on the auto scaling group to let Cluster Autoscaler discover them [1]. Such as:

Key: k8s.io/cluster-autoscaler/node-template/resources/$RESOURCE_NAME
Value: 5
Key: k8s.io/cluster-autoscaler/node-template/label/$LABEL_KEY
Value: $LABEL_VALUE
Key: k8s.io/cluster-autoscaler/node-template/taint/$TAINT_KEY
Value: NoSchedule

Base on current "NodeGroup" construct [2], we could not apply tags from it because managed node group tags currently cannot bring to auto scaling group. Also, "NodeGroup" construct would not return any auto scaling group information which could let us use other function such as "Tags" [3] to directly add tag to auto scaling group.

Currently, eksctl use AWS API at the backend to propagate the managed node group to auto scaling group [4]. I wonder to know could CDK implement the same thing at the backend?

[1] https://docs.aws.amazon.com/eks/latest/userguide/autoscaling.html
[2] https://docs.aws.amazon.com/cdk/api/v1/docs/@aws-cdk_aws-eks.Nodegroup.html
[3] https://docs.aws.amazon.com/cdk/v2/guide/tagging.html
[4] eksctl-io/eksctl#5002

[yws-ss]

@pahud Could you take a look and help reopen this ticket?

[pahud]

Hi @yws-ss

As the original issue is to expose the ASG from the MNG, which is similar but actually not your scenario. Let's create a new issue for that and add me in the loop. Thanks.

[fdhex]

@pahud The initial usecase would still be valid for me, accessing the ASG would be great for full automation otherwise tag management on the ASG are not easily manageable. There might be a workaround looking for the ASG by name and trying to modify it but that should not be the default workflow.

[pahud]

@fdhex
It's possible to write a custom resource to control the ASG behind the MNG.

Consider the following sample:

export interface NodegroupASGModifierProps {
  readonly cluster: eks.ICluster;
  readonly nodegroup: eks.Nodegroup;
  readonly maxInstanceLifetime: Duration;
}

export class NodegroupASGModifier extends Construct {
  constructor(scope: Construct, id: string, props: NodegroupASGModifierProps) {
    super(scope, id);

    const onEventHandler = new lambda.Function(this, 'onEventHandler', {
      handler: 'index.on_event',
      runtime: lambda.Runtime.PYTHON_3_9,
      code: lambda.Code.fromAsset(path.join(__dirname, '../lambda')),
    });

    const nodegroupName = (props.nodegroup.node.defaultChild as CfnNodegroup).getAtt('NodegroupName').toString();
    onEventHandler.addToRolePolicy(new iam.PolicyStatement({
      actions: ['eks:DescribeNodegroup'],
      resources: [
        Stack.of(this).formatArn({
          resource: 'nodegroup',
          service: 'eks',
          resourceName: `${props.cluster.clusterName}/${nodegroupName}/*`,
        }),
      ],
    }));
    onEventHandler.addToRolePolicy(new iam.PolicyStatement({
      actions: ['autoscaling:UpdateAutoScalingGroup'],
      resources: ['*'],
    }));

    const provider = new cr.Provider(this, 'Provider', {
      onEventHandler,
    });


    const myResource = new CustomResource(this, 'CR', {
      serviceToken: provider.serviceToken,
      resourceType: 'Custom::EKSNodegroupModifier',
      properties: {
        clusterName: props.cluster.clusterName,
        nodegroupName,
        maxInstanceLifetime: props.maxInstanceLifetime.toSeconds(),
      },
    });

    const asgName = myResource.getAtt('asg_name').toString();
    new CfnOutput(this, 'ASGName', { value: asgName });

  };
}

With the Lambda handler:

import boto3, json

def update_max_instance_lifetime(asg_name, lifetime):
    client = boto3.client('autoscaling')
    return client.update_auto_scaling_group(
        AutoScalingGroupName=asg_name,
        MaxInstanceLifetime=lifetime
    )

def on_event(event, context):
  print(event)
  request_type = event['RequestType']
  if request_type == 'Create': return on_create(event)
  if request_type == 'Update': return on_update(event)
  if request_type == 'Delete': return on_delete(event)
  raise Exception("Invalid request type: %s" % request_type)

def on_create(event):
  client = boto3.client('eks')  
  props = event["ResourceProperties"]
  print("create new resource with props %s" % props)
  clusterName = props.get('clusterName')
  nodegroupName = props.get('nodegroupName')
  lifetime = props.get('maxInstanceLifetime')
  response = client.describe_nodegroup(
    clusterName=clusterName,
    nodegroupName=nodegroupName,
  )
  asg_name = response['nodegroup']['resources']['autoScalingGroups'][0]['name']
  update_max_instance_lifetime(asg_name, int(lifetime))
  data = { 'asg_name': asg_name }

  return { 'Data': data }

def on_update(event):
    return on_create(event)
#   physical_id = event["PhysicalResourceId"]
#   props = event["ResourceProperties"]
#   print("update resource %s with props %s" % (physical_id, props))
#   # ...

def on_delete(event):
  physical_id = event["PhysicalResourceId"]
  print("delete resource %s" % physical_id)

And create your MNG like this in your stack:

    const vpc = ec2.Vpc.fromLookup(this, 'Vpc', { isDefault: true });
    const cluster = new eks.Cluster(this, 'Cluster', {
      vpc,
      version: eks.KubernetesVersion.V1_23,
      kubectlLayer: new KubectlV23Layer(this, 'KUbectlLayer'),
      defaultCapacity: 0,
    });
    const ng = cluster.addNodegroupCapacity('NG', {
      desiredSize: 2,
    });

    new NodegroupASGModifier(this, 'Modifier', {
      cluster,
      nodegroup: ng,
      maxInstanceLifetime: Duration.days(2),
    });

[zentavr]

@yws-ss hi! Could you please tell me if you were able to resolve your use-case with the tagging the ASG with the tags which the Cluster Autoscaler respects to see?