Running a pod in EKS with Service Accounts, AWS SDK 2

EKS is a brilliant feature from AWS, providing a mostly managed Kubernetes cluster that you can deploy your own services to. If you’re running in AWS you’ll likely be using multiple services such as S3, DynamoDB, SQS, etc. All of these services require credentials to gain access. The best practice is to ensure your code only has permissions to do what it needs to and nothing more, this means using IAM roles. Unfortunately, the documentation is a little confusing and spread out on the subject, so let’s make it as straightforward as possible.

As an important note, this code is somewhat simplified to make it easier to follow. Security best practice has not been followed. Additionally all commands are written as Powershell commands, amend to your shell as required.

Step 1 – Ensure OIDC is configured

The most important thing is that OIDC is enabled in the cluster and connected to AWS IAM. I would strongly recommend that you use the eksctl tool to create your cluster. Ensure you’ve configured IAM in the config file when creating your cluster:

iam:
  withOIDC: true

More detail on a sample initial config file can be found in the eksctl git repo.

 

Step 2 – Create a service account mapped to one or more policies

A service account is like an IAM role, but for EKS specifically.

eksctl create iamserviceaccount --name admin --namespace default --cluster mycluster --attach-policy-arn 'arn:aws:iam::aws:policy/AdministratorAccess' --approve

This will create a service account (role) with the name admin, in the default namespace. In production you should never use this role or access level.

 

Step 3 – Ensure STS is included in your dependencies

The SDKs from AWS are all updated to support EKS service accounts. Sadly, if you’re using v2 of the AWS SDK a critical dependency on STS is missing:

implementation group: 'software.amazon.awssdk', name: 'sts', version: '2.16.17'
implementation group: 'software.amazon.awssdk', name: 'dynamodb', version: '2.16.17'

If you don’t include STS, the SDK will fail to use the service account credentials and fail with various errors.

 

Step 4 – Add the service account to your container spec

apiVersion: batch/v1
kind: Job
metadata:
  name: tagtest
spec:
  template:
    spec:
      serviceAccountName: admin
      containers:
        - name: tagtest
        image: registry/image:version
        imagePullPolicy: Always
      restartPolicy: Never'

 

Done

And that’s it, you’ve got a pod running with a specific set of IAM permissions. You can find more detail on creating service accounts in the eksctl documentation. For production use, you’ll likely want to use the configuration files so you can source control your configuration.

 

Troubleshooting

No region

If the SDK fails with an error referring to the region and lack thereof, the service account hasn’t been set in the container spec.

Unable to load credentials

“software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain”

If you get this error, the STS dependency hasn’t been included in your project. Make sure to manually include it in your build process.

 

General Debugging

The following environment variables are set by the IAM service account provider:

AWS_REGION: eu-west-1
AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
AWS_ROLE_ARN: arn:aws:iam::<accountnumber>:role/<complex name>

If these variables are missing, the service account has not been configured correctly. Check the account has been created:

 
kubectl get serviceaccounts

Also check that the service account is correctly specified in the container spec (see step 4).