kubectl commands don't work

[riontric]

Hi, pipelines are dropped with error:
/usr/bin/kubectl: line 1: syntax error: unexpected redirection
if i use kubectl in command line.
All worked on friday.

[koslib]

Hi, issued release https://github.com/koslib/helm-eks-action/releases/tag/v1.25.3 over the weekend bumping versions of kubectl, awscli and helm. Also tested this out in plenty of my own pipelines and it works okay. Could you post some more details about the structure, jobs and steps of your workflow to take a look?

[riontric]

      - name: Helm deploy
        uses: koslib/helm-eks-action@master
        env:
          KUBE_CONFIG_DATA: ${{ secrets.KUBE_CONFIG_DATA_DEV }}
        with:
          plugins: |
            https://github.com/jkroepke/helm-secrets
          command: |
            aws ecr get-login-password --region us-east-1 | helm registry login --username AWS --password-stdin ${{ env.HELM_PRIVATE_REGISTRY }} && \
              kubectl get pods && helm ls && \
              helm upgrade chart-name -i --atomic -f .env.yaml oci://${{ env.HELM_PRIVATE_REGISTRY }}/helm-chart --version 0.1.9 \
              helm ls && kubectl get pods

this release -> https://github.com/koslib/helm-eks-action/releases/tag/v1.25.2 work for me.

[romanlobai]

Hey, guys! I have the same issue. On Friday lib worked, now - doesn't work.

• name: Update secrets
  uses: koslib/helm-eks-action@master
  with:
  command: |
  kubectl apply -f secrets.yml

error: /usr/bin/kubectl: line 1: syntax error: unexpected redirection

[koslib]

Hmm it might have to do with the fact I started building the base docker image with buildx https://github.com/koslib/helm-eks-action/blob/master/.github/workflows/dockerhub.yaml#L23-L28. Some googling I did indicated this might be a multi-arch issue.

I am contemplating fixing it to linux/amd64 platforms in the build step. Any thoughts or other ideas?

[koslib]

I just run this and built the base docker image with buildx for linux/amd64 platforms as I suspect the multiarch build might have to do something with this.

Could you let me know how it feels now?

[Sharlie89]

Hi, I started to have some errors related to auth API, since you upgrade the docker image (koslib/ga-helm-eks) version from 2.0 to 2.1 & 2.2

If I use koslib/ga-helm-eks:2.0 everything works fine, but if I increase it fails. Have anyone of you experimented this?

The error is related to auth API v1alpha1.

[ruankotovich]

Hello there

The same happening to me today:

      uses: koslib/helm-eks-action@master
      with:
          command: |
            kubectl create secret generic $NAMESPACE-$APP_NAME -n $NAMESPACE \
            --save-config --dry-run=client \
            --from-env-file .env -o yaml | kubectl apply -f -

/usr/bin/kubectl: line 1: syntax error: unexpected redirection

@koslib

[koslib]

@riontric @romanlobai @Sharlie89 I have two potential ideas for this issue:

1. It might be caused of the build arch of the image
2. It might be related to the kubectl version upgrade (although I really don't see how - yet)

Tried updating the koslib/ga-helm-eks:2.2 docker image to in an attempt to address idea no 1. Could you please retry running your actions and see if it has helped or is it still failing with the same command?

To address idea no 2 please mention your cluster versions and we can try a couple of things then.

Thanks in advance for your help 🙏🏼

[koslib]

@koslib guys this is HUGE. please fix this bug ASAP

@SaharAvr We are not a company, I am a person :)

Since I also saw (and kept a copy of) your previous very very rude comment which you deleted, I am prioritizing support for all the kindly-communicating colleagues who reported their issues and actively participate with input in resolving this issue.

And again, this is an open-source repository where you can freely jump in and help and submit a fixing PR. Thanks for your upcoming contribution to this :)

[koslib]

@koslib It was removed because for some reason I mistaken this to be an AWS repo, in which case I wouldn't have removed my original comment

Now that I realise this is a one man's show, I'm going to remove any usages of this action in my workplace, as this is highly dangerous

Feel free 👋🏼

[Sharlie89]

Hey @koslib,

I'm gonna test it today with the image 2.2. I've keep this working with image 2.0 but with the new versions of the kubectl, awscli and helm. I got a warning about the AWS api v1alpha1 (when it's supposed to be v1beta1), but hope this give you a clue about what is happening (I also thought that is related to the build arch).

@SaharAvr What's is highly dangerous is to have someone on charge of something who's implementing things that he's no testing and understanding at all. Also, goes to public github repo, demanding disciplinary treatment is, from my point of view, a clear behaviour of a bad person. EVERYONE OF US MAKES MISTAKES, everyone. You can't go to a public site demanding punishment for one error. Also this is a free use action, so I don't know why you feel in the place for demanding this kind of stuff.

[Sharlie89]

@koslib

Just tested it, doesn't working at all. I'm getting again this error:

Error: Kubernetes cluster unreachable: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1alpha1"

I doesn't know why, but this is happening in one of my GH actions, but it's working on another. Both of them had the same cluster as an endpoint.

[koslib]

@Sharlie89 well, I think I understand what's up - let me explain and please let me know if this makes any sense to you too:

• I issued version v1.25.3 which included a helm version bump to 3.10.1.
• However, it seems there is a bug with recent input as well mentioning that helm 3.9.x broke stuff for EKS clusters
• A workaround recommended is to downgrade to helm 3.8.2 but I don't like this solution as this is a step backwards..
• I'll experiment a bit with upgrading to awscli v2 which presumably fixes this error

I'll get back to you as I have more updates - hopefully before the weekend! Your time and help is really appreciated 🙏🏼

[koslib]

@Sharlie89 I was actually wrong. Turns out the issue has to do with the awscli@1.24.x version, as k8s upgraded the auth api and the token given from that aws cli version did not support that correctly.

I made a new branch named experimental where:

• removed the now obsolete aws-iam-authenticator
• upgraded awscli and pinned version to 1.25.96, which seems to have this problem fixed

and tried this out with no issues on my end.

Could you give it a shot as well to confirm it's also ok for your systems before I PR to master?

- uses: koslib/helm-eks-action@experimental
...

Potentially @riontric and @romanlobai can try this out too and see how that goes?

Thank you all in advance!

[Sharlie89]

@koslib just tested it, same error.
Error: Kubernetes cluster unreachable: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1alpha1"
At this point I don't know what is happening. Also I don't know why I'm getting and invalid apiVersion with v1alpha1, when Kubeconfig is pointing to v1beta1. But more weird is the fact that it's works on one job and failing in another.

[Sharlie89]

Looking through all of this, the only pain point that I see is that all of this started when you changed the way of built the docker image. But this is what I see from my site, can't be sure 100%.
I forked your repo when you where setting the docker image koslib/ga-helm-eks:2.0, and this works for me. But not any of the other images above 2.0.

I updated one of my clusters today to 1.22 versions, but no luck also. I ran out of ideas about what can be happening.

[koslib]

I'm trying to piggy back on your train of thought @Sharlie89 .

• 2.0 brought the sops executable in the base image, and this is the last image known to work on your end
• In this PR appears to have derailed things.

What this PR did:

• Updated base image Dockerfile to include awscli==1.24.0, kubectl==1.25.91 and helm==3.10.1
• Included a manual-trigger github action to build the base image and push it to dockerhub. This action builds the image using buildx and should support multi-arch builds.
• Changed base image to tag 2.1 (which included the changes above ^^)

In order to eliminate any multi-arch issues later on, I built and pushed locally with --platform linux/amd64, but the issue persisted. Which leaves us with the dependencies updates.

just tested it, same error.
Error: Kubernetes cluster unreachable: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1alpha1"
At this point I don't know what is happening. Also I don't know why I'm getting and invalid apiVersion with v1alpha1, when Kubeconfig is pointing to v1beta1

In the experimental branch you got this error, which eliminates eg. an outdated aws-iam-authenticator to be the root cause. Therefore maybe a difference between the awscli of the action and your version for generating the kube context data? Can you post more details regarding how you generate your kubeconfig file?

But more weird is the fact that it's works on one job and failing in another.

Could you post how these jobs look like? There must be a difference somewhere.

Finally, you have an EKS at v1.22, right? I'll build a new test cluster mimicking exactly your setup to test this out. Updates to come.

[Sharlie89]

In this one, is working:

name: Deploy to Sandbox

env:
  AWS_REGION: eu-central-1

on:
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout                                  
      uses: actions/checkout@v2                       
      with:                                           
        ref: ${{ github.event.pull_request.head.sha }}
        
    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.SBX_AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.SBX_AWS_SECRET_ACCESS_KEY }}
        aws-region: ${{ env.AWS_REGION }}

    - name: Login to Amazon ECR
      id: login-ecr
      uses: aws-actions/amazon-ecr-login@v1

    - name: Build, tag, and push image to Amazon ECR
      env:
        ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
        ECR_REPOSITORY: api-gateway-sandbox
        IMAGE_TAG: ${{ github.sha }}
      run: |
        docker build --build-arg BASE_IMAGE=node:17.6.0-alpine -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .
        docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
    - name: Deploy to Cluster
      uses: koslib/helm-eks-action@master
      env:
        KUBE_CONFIG_DATA: ${{ secrets.SBX_KUBECONFIG }}
      with:
        command: helm upgrade api-gateway --install ./helm/api-gateway --wait --set image.tag=${{ github.sha }} -f ./helm/values/sandbox/values.yaml -n XXXXX

  slack_notification:
    needs: [build]
    if: failure()
    uses: XXXXXX/infra-api-gateway/.github/workflows/slack_notification.yml@develop
    secrets:
      SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

In this, fails:

  name: Deploy to Sandbox

env:
  AWS_REGION: eu-central-1

on:
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout                                  
      uses: actions/checkout@v2                       
      with:                                           
        ref: ${{ github.event.pull_request.head.sha }}
        
    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.SBX_AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.SBX_AWS_SECRET_ACCESS_KEY }}
        aws-region: ${{ env.AWS_REGION }}

    - name: Login to Amazon ECR
      id: login-ecr
      uses: aws-actions/amazon-ecr-login@v1

    - name: Build, tag, and push image to Amazon ECR
      env:
        ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
        ECR_REPOSITORY: monolith-sandbox
        IMAGE_TAG: ${{ github.sha }}
      run: |
        docker build --build-arg BASE_IMAGE=node:17.6.0-alpine -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .
        docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
    - name: Build development image, tag, and push image to Amazon ECR
      env:
        ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
        ECR_REPOSITORY: monolith-sandbox
        IMAGE_TAG: ${{ github.sha }}
      run: |
        docker build -f Dockerfile.migrations --build-arg BASE_IMAGE=node:17.6.0-alpine -t ${ECR_REGISTRY}/${ECR_REPOSITORY}:${IMAGE_TAG}-development .
        docker push ${ECR_REGISTRY}/${ECR_REPOSITORY}:${IMAGE_TAG}-development
    - name: Deploy to Cluster
      uses: koslib/helm-eks-action@experimental
      env:
        KUBE_CONFIG_DATA: ${{ secrets.SBX_KUBECONFIG }}
      with:
        command: helm upgrade monolith --install ./helm/monolith --wait --timeout 10m --set image.tag=${{ github.sha }} -f ./helm/values/sandbox/values.yaml -n XXXXX

  slack_notification:
    needs: [build]
    if: failure()
    uses: XXXX/infra-monolith/.github/workflows/slack_notification.yml@develop
    secrets:
      SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

[koslib]

So the master branch works for you but not the experimental one. Does this also mean tag v1.25.4 works? 🤔 Now I'm getting more confused :D But I'll try to find some time during the coming days to try stuff out on demo clusters.

[Sharlie89]

So the master branch works for you but not the experimental one. Does this also mean tag v1.25.4 works? 🤔 Now I'm getting more confused :D But I'll try to find some time during the coming days to try stuff out on demo clusters.

No no, in the failing job doesn't work with: experimental, master or v1.25.4. Only works with the docker image koslib/ga-helm-eks:2.0.

I've tested this with my forked repo.