NextChapterSoftware/ec2-action-builder

customTags is not iterable and instance doesn't terminate on failure

Closed this issue · 7 comments

MaxDiOrio commented

Working on using this and looks promising.

When testing, I get:

/home/runner/work/_actions/NextChapterSoftware/ec2-action-builder/v1.8/dist/index.js:176
...customTags
^

TypeError: customTags is not iterable
at Ec2Instance.getTags (/home/runner/work/_actions/NextChapterSoftware/ec2-action-builder/v1.8/dist/index.js:176:16)
at new Ec2Instance (/home/runner/work/_actions/NextChapterSoftware/ec2-action-builder/v1.8/dist/index.js:134:26)
at /home/runner/work/_actions/NextChapterSoftware/ec2-action-builder/v1.8/dist/index.js:1016:27
at Generator.next ()
at /home/runner/work/_actions/NextChapterSoftware/ec2-action-builder/v1.8/dist/index.js:1000:71
at new Promise ()
at webpack_modules.3109.__awaiter (/home/runner/work/_actions/NextChapterSoftware/ec2-action-builder/v1.8/dist/index.js:996:12)
at start (/home/runner/work/_actions/NextChapterSoftware/ec2-action-builder/v1.8/dist/index.js:1014:12)
at /home/runner/work/_actions/NextChapterSoftware/ec2-action-builder/v1.8/dist/index.js:1093:13
at Generator.next ()

I was randomly able to get it to run once, even though nothing changed oddly enough. That one run failed with a timeout and the EC2 instance was not cleaned up - it remains running.

mahdi-torabi commented

Hey, Could you share the GH actions yaml so I could see what's going on with the tags ?

MaxDiOrio commented

Pretty straight forward. Same thing happens if I define tags in the actions yaml.

name: ec2-runner
on: pull_request
jobs:
    start-runner:
        timeout-minutes: 5                  # normally it only takes 1-2 minutes
        name: Start self-hosted EC2 runner   
        runs-on: ubuntu-latest
        permissions:
          actions: write        
        steps:      
          - name: Start EC2 runner
            id: start-ec2-runner
            uses: NextChapterSoftware/ec2-action-builder@v1.8
            with:
              github_token: ${{ secrets.GH_RUNNER_PAT }}
              aws_access_key_id: ${{ secrets.AWS_ACCESS_KEY_ID }}
              aws_secret_access_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
              aws_region: ${{ secrets.AWS_REGION }}
              github_job_start_ttl_seconds: 60                        # Optional - (defaults to 0 disabling this feature)
              github_api_retry_delay: 10                              # Optional - Delay when polling for runner registration (default is 10 seconds)
              ec2_instance_type: t3.nano
              ec2_ami_id: ami-085f9c64a9b75eed5            #Ubuntu 24.04 LTS Stock Image
              # ec2_root_disk_size_gb: "100"               # Optional - (defaults to AMI settings)
              # ec2_root_disk_ebs_class: "gp2"             # Optional - Only used with custom volume root size (defaults to gp2)
              ec2_subnet_id: "subnet-redacted"    #github-runners - this does not change
              ec2_security_group_id: "sg-redacted" #github-runners-default, no inbound, 80/443 outbound
              ec2_instance_ttl: 40                          # Optional - (default is 60 minutes)
              ec2_spot_instance_strategy: None    # Other options are: None, BestEffort, MaxPerformance 
    # Job that runs on the self-hosted runner 
    run-build:
        timeout-minutes: 5
        needs:
          - start-runner
        runs-on: ${{ github.run_id }}          
        steps: 
          - name: Sample
            run: env
          - name: Install dependencies
            run: |
              sudo apt-get update -y
              sudo apt-get upgrade -y
              sudo install -m 0755 -d /etc/apt/keyrings
              sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
              sudo chmod a+r /etc/apt/keyrings/docker.asc
              echo \
                "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
                $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
                sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
              sudo apt-get update -y
              sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
          - name: Run Container
            run: sudo docker run hello-world
mahdi-torabi commented

Seems like a regression. We had similar issues in the past which we fixed:

Let me try and see if I can reproduce it. I'll keep you posted

mahdi-torabi commented

Fix PR merged: #37

Here's the new release: https://github.com/NextChapterSoftware/ec2-action-builder/releases/tag/v1.9

Use version v1.9 and it should work as expected.

MaxDiOrio commented

Perfect - works great now. Thanks.

Just to clarify - what happens when a build fails and the EC2 instance doesn't run the shutdown script? Twice I've had a runner job fail, but the EC2 instance is never cleaned up. The one below was a timeout waiting for the self-hosted runner to register.

Ec2 spot instance strategy is set to none
Starting instance with none strategy
AWS EC2 instance i-01967a62320981c42 is up and running
Waiting 30s before polling for runner
Polling for runner every 10s
Waiting...
Waiting...
...
Waiting...
Error: The operation was canceled.

And the instance remained up.

mahdi-torabi commented

Could you open a separate issue for that. I have noticed a couple of these as well. Instance TTL eventually takes care of them (default is 60 mins I believe) but we also added a registration timeout so that if the instance is idle more than two minutes it should be torn down.

MaxDiOrio commented

Done. Thanks for the help!