Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(cli): (cdk fails with "Profile <profile name> did not include credential process) #25870

Open
kirnberger1980 opened this issue Jun 6, 2023 · 10 comments · May be fixed by #31702
Open

(cli): (cdk fails with "Profile <profile name> did not include credential process) #25870

kirnberger1980 opened this issue Jun 6, 2023 · 10 comments · May be fixed by #31702
Assignees
Labels
bug This issue is a bug. effort/medium Medium work item – several days of effort p1 package/tools Related to AWS CDK Tools or CLI

Comments

@kirnberger1980
Copy link

kirnberger1980 commented Jun 6, 2023

Describe the bug

We want to use the CDK CLI with Gitlab Runner, which is running on EKS in Account A. cdk deploy is executed for Account B (Cross-Account Deployment).

cdk deploy --profile AccountB

CLI Command fails with:

Unable to determine the default AWS account (ProcessCredentialsProviderFailure): Profile <AccountB> did not include credential process
....
Could not assume arn:aws:iam::xxxxxxx:role/cdk-xxxxxxxx-lookup-role-xxxxxxx-eu-central-1, proceeding anyway.

We are using IRSA (IAM Role for Service Accounts), further called Role A, for the Gitlab Runner in Account A.
In Account B there is an IAM Role (Role B), which can be assumed by Role A. RoleB can assume all cdk-provisioned IAM Roles in that Account (created by cdk bootstrap).

Profile file (stored in /configmaps/awsconfig):

    [default]
    region = eu-central-1
    role_arn = arn:aws:iam::AccountA:role/RoleA
    web_identity_token_file=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
    output = json


    [profile AccountB]
    role_arn = arn:aws:iam::AccountB:role/RoleB
    source_profile = default
    output = json
    region = eu-central-1

We have set environment variables AWS_PROFILE and AWS_CONFIG_FILE:

AWS_PROFILE=AccountB
AWS_CONFIG_FILE=/configmaps/awsconfig

AWS CLI is working fine with above profile file and the environment variables:

aws sts get-caller-identity
{
    "UserId": "AROA2SFT2XXXXXXXXX:botocore-session-168600000",
    "Account": "00000000000",
    "Arn": "arn:aws:sts::00000000000:assumed-role/RoleB/botocore-session-168600000"
}

Assuming the cdk-lookup Role from RoleB with AWS CLI works fine.

aws sts assume-role --role-arn arn:aws:iam::xxxxxxx:role/cdk-xxxxxxxx-lookup-role-xxxxxxx-eu-central-1 --role-session-name test123
{
    "Credentials": {
        "AccessKeyId": "<somekey>",
        "SecretAccessKey": "<somesecret>",
        "SessionToken": "<nice jwt>",
        "Expiration": "2023-06-06T15:29:36+00:00"
    },
    "AssumedRoleUser": {
        "AssumedRoleId": "<some otherkey:test123",
        "Arn": "arn:aws:iam::xxxxxxx:role/cdk-xxxxxxxx-lookup-role-xxxxxxx-eu-central-1"
    }
}

By the way: cdk also seems not to respect AWS_PROFILE environment variable as it is using the default role (it could be, that the session token by IRSA is used), when NOT using --profile.

Expected Behavior

We expect that AWS CDK CLI is acting identically to AWS CLI when using AWS profiles for Cross-Account Deployments. Environment Variable AWS_PROFILE should be respected.

Current Behavior

The CDK CLI fails that the cdk-Lookup-Role cannot be assumed, because "Profile did not include credential process". When you don't specify --profile, then default-Profile (or session token created by OIDC Provider of IAM) is used.

Reproduction Steps

You need to run a pod in EKS:

  • configure IRSA (RoleA) for the pod in the same account (Account A)
  • configure a RoleB in Account B, which can be assumed by RoleA
  • give permissions to RoleB to assume cdk-provisioned roles
  • configure an AWS_CONFIG_FILE like described above.
  • install CDK, NODE etc on the pod
  • Configure env variables AWS_PROFILE and AWS_CONFIG_FILE
  • Run cdk deploy

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.78.0

Framework Version

2.78.0

Node.js Version

v18.16.0

OS

node:18-alpine

Language

Typescript

Language Version

3.9.7

Other information

No response

@kirnberger1980 kirnberger1980 added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jun 6, 2023
@github-actions github-actions bot added the package/tools Related to AWS CDK Tools or CLI label Jun 6, 2023
@pahud
Copy link
Contributor

pahud commented Jun 6, 2023

Thanks for reporting.

  1. Can you run this in the pod from Account A?
aws --profile AccountB sts get-caller-identity
  1. Can you share a minimal CDK code that provisions this environment so I can try reproduce it in my account?

@pahud pahud self-assigned this Jun 6, 2023
@pahud pahud added investigating This issue is being investigated and/or work is in progress to resolve the issue. p2 effort/medium Medium work item – several days of effort response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. and removed needs-triage This issue or PR still needs to be triaged. labels Jun 6, 2023
@kirnberger1980
Copy link
Author

kirnberger1980 commented Jun 7, 2023

Thanks for the quick reply.

  1. Here is the result of the aws cli response:
{
    "UserId": "AXXXXXXXXXX:botocore-session-1686146634",
    "Account": "<AccountB>",
    "Arn": "arn:aws:sts::<AccountB>:assumed-role/RoleB/botocore-session-1686146634"
}

This is, what we expected, as setting env variable AWS_PROFILE or using --profile should return the identical response.

  1. Here is some sample code tested, where we could reproduce the issue:

bin-File:

#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';
import { CdkStack } from '../lib/cdk-stack';

const app = new cdk.App();
new CdkStack(app, 'CdkStack', {
  env: { account: "<AccountB>",region: "eu-central-1" }
});

lib-File:

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';

export class CdkStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

  const vpc = cdk.aws_ec2.Vpc.fromLookup(this, "MyVPC", {
    vpcName: "MyVpcName"
  });

  new cdk.aws_ec2.SecurityGroup(this, "MyNewSecurityGroup", {
    vpc,
    description: "This will secure my resources"
  });
  }
}

Kind Regards,
Marcell

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Jun 7, 2023
@kirnberger1980
Copy link
Author

Any news on this issue?

@github-actions github-actions bot added p1 and removed p2 labels Jan 28, 2024
Copy link

This issue has received a significant amount of attention so we are automatically upgrading its priority. A member of the community will see the re-prioritization and provide an update on the issue.

@jumpinjan
Copy link

I am facing a similar issue. An update would be greatly appreciated.

@elixxx
Copy link

elixxx commented Feb 8, 2024

Whats going on with this issue? We are facing the same issue but sounds like a common workflow!

@megakoresh
Copy link

megakoresh commented Feb 19, 2024

Is this a recent thing? I just now saw this. CDK worked totally fine then suddenly it's now failing with this error and in the verbose log I see profile xxx did not include credential process. I have the profile name exported in the AWS_PROFILE variable. I reset the terminal and ran aws sso login again, but nothing helped. Removal of cdk.out folda also didn't help. It's as if it broke as a result of some change in the cloud since it broke today after 16:00 and before that worked just fine.

UPDATE: I managed to fix this by updating the awscliv2 to the latest version, doing aws sso logout then aws sso login and then it worked. So might be something to do with token timings, and not CDK per se. Still, strange. AWS CLI commands worked just fine.

@megakoresh
Copy link

Updating: this seems to have something to do with timing. This happens if the session expires. Once it does, CDK will break and no amount of logging out and back in will help. Only waiting seems to do the trick. I have also tried clearing context, but still nothing. After waiting for hours, and then logging out and back in the problem goes away until the next day, but that is hardly a "solution". Is anyone looking into this?

@sturman
Copy link

sturman commented Apr 4, 2024

I think the problem is in web_identity_token_file.
In my case with AWS_PROFILE=oidc environment variable set and the following content in ~/.aws/config file

[profile oidc]
role_arn=arn:aws:iam::012345678901:role/gitlab-deploy
web_identity_token_file=/tmp/web_identity_token

command aws sts get-caller-identity works as expected.
But npx cdk diff -vv fails with

...
[16:42:45] Determining if we're on an EC2 instance.
[16:42:45] Does not look like an EC2 instance.
[16:42:45] Toolkit stack: CDKToolkit
[16:42:45] Setting "CDK_DEFAULT_REGION" environment variable to eu-central-1
[16:42:45] Resolving default credentials
[16:42:45] Unable to determine the default AWS account (ProcessCredentialsProviderFailure): Profile oidc did not include credential process
[16:42:45] context: {
...

So I ended up assuming role and setting AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN environment variables with command

export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s"
$(aws sts assume-role-with-web-identity
--role-arn ${ROLE_ARN}
--role-session-name "GitLabRunner-${CI_PROJECT_ID}-${CI_PIPELINE_ID}"
--web-identity-token ${GITLAB_OIDC_TOKEN}
--duration-seconds 3600
--query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]'
--output text))

@pahud pahud removed their assignment Aug 7, 2024
@pahud pahud removed the investigating This issue is being investigated and/or work is in progress to resolve the issue. label Aug 7, 2024
@otaviomacedo otaviomacedo self-assigned this Oct 2, 2024
@TheRealAmazonKendra
Copy link
Contributor

I'm pretty sure the issue here is that our current credential chain does not support web_identity_token_file inside an ini file, so the credential chain here fails on profileCredentials because of the presence of web_identity_token_file and then again on SsoCredentials because none are setup, as expected. The last credential provider is ProcessCredentials, which will fail here, also as expected. Since it is the last in the chain, that's the error that actually gets surfaced, which is super misleading.

We are currently in the process of upgrading from sdkv2 to sdkv3, at which point web_identity_token_file will be supported and you should no longer experience this error. You can follow the progress in #31702, if you'd like.

@TheRealAmazonKendra TheRealAmazonKendra self-assigned this Oct 10, 2024
@TheRealAmazonKendra TheRealAmazonKendra linked a pull request Oct 15, 2024 that will close this issue
28 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug. effort/medium Medium work item – several days of effort p1 package/tools Related to AWS CDK Tools or CLI
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants