Following start guide - how to view created resources in AWS console?
andrew89898 opened this issue · 15 comments
Apologies for the newbie question - I am learning about Agones, GameLift FleetIQ and plan to use AWS FlexMatch to create a game service for CSGO (source engine) - I plan to use a webhook autoscaler with AWS and write a wrapper for the game server.
I followed the guide to create a workspace and set up all the aws resources - all good managed to get to the end of the guide and proceed with the scaling up and replicas seeing the nodes - however it didn't ever scale up to 100 nodes stopping at 54 with 1 group and 60 with 2 groups. I'd like to understand more of this adapter and agones but I can't see any of these resources in my AWS account except for the GameLift groups. The EKS cluster does show up but doesn't display any nodes.
This isn't really an issue with the repo - I'd just like to learn more but I can't see any forum or slack channel/similar for this. I'd like to be able to work on the components of the solution architecture and visualise where I need to work on, if you guys can give any recommendations.
Thanks.
@andrew89898 you may not have permission to view the nodes in your EKS cluster from the EKS console. Can you run kubectl get nodes and add the output from that command to this issue? If you've installed the Kubernetes cluster autoscaler, please include the logs from that pod too, e.g. kubectl logs <podname> -n <namespace>. I'd also like to know how many instances in the ASG (if any) are in a standby state.
Sure can do:
NAME STATUS ROLES AGE VERSION
ip-192-168-2-176.eu-west-2.compute.internal Ready <none> 3d22h v1.18.9-eks-d1db3c
ip-192-168-23-13.eu-west-2.compute.internal Ready <none> 3d20h v1.18.9-eks-d1db3c
ip-192-168-41-125.eu-west-2.compute.internal Ready <none> 6h20m v1.18.9-eks-d1db3c
ip-192-168-60-223.eu-west-2.compute.internal Ready <none> 3d22h v1.18.9-eks-d1db3c
ec2-user:~/environment $
Logs (lots of lines, got first page):
(command is: kubectl logs cluster-autoscaler-57bdc587bf-v66tp -n kube-system
E0511 17:45:00.878519 1 static_autoscaler.go:268] Failed to get node infos for groups: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
I0511 17:45:04.979515 1 reflector.go:496] k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:246: Watch close - *v1.Node total 7 items received
I0511 17:45:10.878649 1 static_autoscaler.go:226] Starting main loop
E0511 17:45:10.878891 1 utils.go:119] Unable to build proper template node for gamelift-gameservergroup-: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
E0511 17:45:10.878903 1 static_autoscaler.go:268] Failed to get node infos for groups: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
I0511 17:45:20.879035 1 static_autoscaler.go:226] Starting main loop
E0511 17:45:20.879272 1 utils.go:119] Unable to build proper template node for gamelift-gameservergroup-: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
E0511 17:45:20.879284 1 static_autoscaler.go:268] Failed to get node infos for groups: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
I0511 17:45:26.996389 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache
E0511 17:45:26.996420 1 node_instances_cache.go:164] Failed to get cloud provider node instance for node group gamelift-gameservergroup-, error error while looking for instances of ASG: {gamelift-gameservergroup-}
I0511 17:45:26.996428 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 18.977µs
I0511 17:45:30.879414 1 static_autoscaler.go:226] Starting main loop
E0511 17:45:30.879661 1 utils.go:119] Unable to build proper template node for gamelift-gameservergroup-: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
E0511 17:45:30.879674 1 static_autoscaler.go:268] Failed to get node infos for groups: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
I0511 17:45:37.179538 1 reflector.go:496] k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:309: Watch close - *v1beta1.PodDisruptionBudget total 0 items received
I0511 17:45:40.879802 1 static_autoscaler.go:226] Starting main loop
E0511 17:45:40.880032 1 utils.go:119] Unable to build proper template node for gamelift-gameservergroup-: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
E0511 17:45:40.880043 1 static_autoscaler.go:268] Failed to get node infos for groups: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
I0511 17:45:50.880186 1 static_autoscaler.go:226] Starting main loop
E0511 17:45:50.880445 1 utils.go:119] Unable to build proper template node for gamelift-gameservergroup-: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
E0511 17:45:50.880458 1 static_autoscaler.go:268] Failed to get node infos for groups: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
I0511 17:46:00.880606 1 static_autoscaler.go:226] Starting main loop
I0511 17:46:00.880659 1 auto_scaling_groups.go:351] Regenerating instance to ASG map for ASGs: [gamelift-gameservergroup- gamelift-gameservergroup-agones-game-server-group-02]
I0511 17:46:00.942462 1 auto_scaling.go:199] 1 launch configurations already in cache
I0511 17:46:00.942486 1 auto_scaling_groups.go:114] Updating ASG gamelift-gameservergroup-agones-game-server-group-02
I0511 17:46:00.942495 1 aws_manager.go:269] Refreshed ASG list, next refresh after 2021-05-11 17:47:00.942492397 +0000 UTC m=+17643.158739587
E0511 17:46:00.942694 1 utils.go:119] Unable to build proper template node for gamelift-gameservergroup-: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
E0511 17:46:00.942705 1 static_autoscaler.go:268] Failed to get node infos for groups: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
I0511 17:46:10.942867 1 static_autoscaler.go:226] Starting main loop
E0511 17:46:10.943115 1 utils.go:119] Unable to build proper template node for gamelift-gameservergroup-: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
E0511 17:46:10.943126 1 static_autoscaler.go:268] Failed to get node infos for groups: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
I0511 17:46:13.683475 1 reflector.go:496] k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:212: Watch close - *v1.Pod total 4 items received
I0511 17:46:20.943283 1 static_autoscaler.go:226] Starting main loop
E0511 17:46:20.943537 1 utils.go:119] Unable to build proper template node for gamelift-gameservergroup-: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
E0511 17:46:20.943548 1 static_autoscaler.go:268] Failed to get node infos for groups: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
I0511 17:46:30.943707 1 static_autoscaler.go:226] Starting main loop
E0511 17:46:30.943952 1 utils.go:119] Unable to build proper template node for gamelift-gameservergroup-: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
E0511 17:46:30.943964 1 static_autoscaler.go:268] Failed to get node infos for groups: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
I0511 17:46:40.944109 1 static_autoscaler.go:226] Starting main loop
E0511 17:46:40.944343 1 utils.go:119] Unable to build proper template node for gamelift-gameservergroup-: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
E0511 17:46:40.944354 1 static_autoscaler.go:268] Failed to get node infos for groups: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
I0511 17:46:44.389064 1 reflector.go:496] k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:320: Watch close - *v1.DaemonSet total 3 items received
I0511 17:46:50.944493 1 static_autoscaler.go:226] Starting main loop
E0511 17:46:50.944732 1 utils.go:119] Unable to build proper template node for gamelift-gameservergroup-: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
E0511 17:46:50.944744 1 static_autoscaler.go:268] Failed to get node infos for groups: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
I0511 17:47:00.944886 1 static_autoscaler.go:226] Starting main loop
I0511 17:47:00.944932 1 auto_scaling_groups.go:351] Regenerating instance to ASG map for ASGs: [gamelift-gameservergroup-agones-game-server-group-02 gamelift-gameservergroup-]
I0511 17:47:01.039568 1 auto_scaling.go:199] 1 launch configurations already in cache
I0511 17:47:01.039592 1 auto_scaling_groups.go:114] Updating ASG gamelift-gameservergroup-agones-game-server-group-02
I0511 17:47:01.039600 1 aws_manager.go:269] Refreshed ASG list, next refresh after 2021-05-11 17:48:01.039597629 +0000 UTC m=+17703.255844816
E0511 17:47:01.039844 1 utils.go:119] Unable to build proper template node for gamelift-gameservergroup-: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
E0511 17:47:01.039856 1 static_autoscaler.go:268] Failed to get node infos for groups: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
I0511 17:47:04.380433 1 reflector.go:496] k8s.io/client-go/informers/factory.go:135: Watch close - *v1.PersistentVolume total 0 items received
I0511 17:47:11.040029 1 static_autoscaler.go:226] Starting main loop
E0511 17:47:11.040299 1 utils.go:119] Unable to build proper template node for gamelift-gameservergroup-: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
E0511 17:47:11.040312 1 static_autoscaler.go:268] Failed to get node infos for groups: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
I0511 17:47:21.040466 1 static_autoscaler.go:226] Starting main loop
E0511 17:47:21.040714 1 utils.go:119] Unable to build proper template node for gamelift-gameservergroup-: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
E0511 17:47:21.040726 1 static_autoscaler.go:268] Failed to get node infos for groups: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
I0511 17:47:26.996557 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache
E0511 17:47:26.996589 1 node_instances_cache.go:164] Failed to get cloud provider node instance for node group gamelift-gameservergroup-, error error while looking for instances of ASG: {gamelift-gameservergroup-}
I0511 17:47:26.996598 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 22.646µs
I0511 17:47:31.040886 1 static_autoscaler.go:226] Starting main loop
E0511 17:47:31.041132 1 utils.go:119] Unable to build proper template node for gamelift-gameservergroup-: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
E0511 17:47:31.041144 1 static_autoscaler.go:268] Failed to get node infos for groups: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
I0511 17:47:41.041290 1 static_autoscaler.go:226] Starting main loop
E0511 17:47:41.041529 1 utils.go:119] Unable to build proper template node for gamelift-gameservergroup-: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
E0511 17:47:41.041541 1 static_autoscaler.go:268] Failed to get node infos for groups: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
I0511 17:47:51.041679 1 static_autoscaler.go:226] Starting main loop
E0511 17:47:51.041925 1 utils.go:119] Unable to build proper template node for gamelift-gameservergroup-: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
E0511 17:47:51.041936 1 static_autoscaler.go:268] Failed to get node infos for groups: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
I0511 17:48:01.042324 1 static_autoscaler.go:226] Starting main loop
I0511 17:48:01.042393 1 auto_scaling_groups.go:351] Regenerating instance to ASG map for ASGs: [gamelift-gameservergroup-agones-game-server-group-02 gamelift-gameservergroup-]
I0511 17:48:01.199847 1 auto_scaling.go:199] 1 launch configurations already in cache
I0511 17:48:01.199873 1 auto_scaling_groups.go:114] Updating ASG gamelift-gameservergroup-agones-game-server-group-02
I0511 17:48:01.199883 1 aws_manager.go:269] Refreshed ASG list, next refresh after 2021-05-11 17:49:01.199880066 +0000 UTC m=+17763.416127279
E0511 17:48:01.200090 1 utils.go:119] Unable to build proper template node for gamelift-gameservergroup-: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
E0511 17:48:01.200103 1 static_autoscaler.go:268] Failed to get node infos for groups: unable to get first AvailabilityZone for ASG "gamelift-gameservergroup-"
ec2-user:~/environment $
There may be an issue with how you've configured cluster-autoscaler. Can you run kubectl get pod cluster-autoscaler-57bdc587bf-v66tp -n kube-system and paste the output here.
You have 2 ASG/Game Server Groups? One is called gamelift-gameservergroup-agones-game-server-group-2 and the other is called gamelift-gameservergroup-? There are a multitude of things you need to do if you want to use the adapter with multiple game server groups. If you don't do it properly, it may not work.
Yeah I just followed the full guide from the repo - right to the end including creating the second group. Not sure if I even need 2 groups. Those look like the right names not sure how the first group got trimmed to end on a -.
Probably need to fix that somewhere.
ec2-user:~/environment $ kubectl get pod cluster-autoscaler-57bdc587bf-v66tp -n kube-system
NAME READY STATUS RESTARTS AGE
cluster-autoscaler-57bdc587bf-v66tp 1/1 Running 0 5h56m
ec2-user:~/environment $
can you add -o yaml to the end of that command? I need to see the details. Thanks!
Yeah got it seems to be decent amount of yaml:
ec2-user:~/environment $ kubectl get pod cluster-autoscaler-57bdc587bf-v66tp -n kube-system -o yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
kubernetes.io/psp: eks.privileged
prometheus.io/port: "8085"
prometheus.io/scrape: "true"
creationTimestamp: "2021-05-11T12:52:55Z"
generateName: cluster-autoscaler-57bdc587bf-
labels:
app: cluster-autoscaler
pod-template-hash: 57bdc587bf
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:prometheus.io/port: {}
f:prometheus.io/scrape: {}
f:generateName: {}
f:labels:
.: {}
f:app: {}
f:pod-template-hash: {}
f:ownerReferences:
.: {}
k:{"uid":"11ce4443-49b2-482a-a50a-acd33b0c0048"}:
.: {}
f:apiVersion: {}
f:blockOwnerDeletion: {}
f:controller: {}
f:kind: {}
f:name: {}
f:uid: {}
f:spec:
f:containers:
k:{"name":"cluster-autoscaler"}:
.: {}
f:command: {}
f:env:
.: {}
k:{"name":"AWS_REGION"}:
.: {}
f:name: {}
f:value: {}
f:image: {}
f:imagePullPolicy: {}
f:name: {}
f:resources:
.: {}
f:limits:
.: {}
f:cpu: {}
f:memory: {}
f:requests:
.: {}
f:cpu: {}
f:memory: {}
f:terminationMessagePath: {}
f:terminationMessagePolicy: {}
f:volumeMounts:
.: {}
k:{"mountPath":"/etc/ssl/certs/ca-certificates.crt"}:
.: {}
f:mountPath: {}
f:name: {}
f:readOnly: {}
f:dnsPolicy: {}
f:enableServiceLinks: {}
f:restartPolicy: {}
f:schedulerName: {}
f:securityContext: {}
f:serviceAccount: {}
f:serviceAccountName: {}
f:terminationGracePeriodSeconds: {}
f:volumes:
.: {}
k:{"name":"ssl-certs"}:
.: {}
f:hostPath:
.: {}
f:path: {}
f:type: {}
f:name: {}
manager: kube-controller-manager
operation: Update
time: "2021-05-11T12:52:55Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:status:
f:conditions:
k:{"type":"ContainersReady"}:
.: {}
f:lastProbeTime: {}
f:lastTransitionTime: {}
f:status: {}
f:type: {}
k:{"type":"Initialized"}:
.: {}
f:lastProbeTime: {}
f:lastTransitionTime: {}
f:status: {}
f:type: {}
k:{"type":"Ready"}:
.: {}
f:lastProbeTime: {}
f:lastTransitionTime: {}
f:status: {}
f:type: {}
f:containerStatuses: {}
f:hostIP: {}
f:phase: {}
f:podIP: {}
f:podIPs:
.: {}
k:{"ip":"192.168.35.195"}:
.: {}
f:ip: {}
f:startTime: {}
manager: kubelet
operation: Update
time: "2021-05-11T12:52:58Z"
name: cluster-autoscaler-57bdc587bf-v66tp
namespace: kube-system
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: cluster-autoscaler-57bdc587bf
uid: 11ce4443-49b2-482a-a50a-acd33b0c0048
resourceVersion: "1185778"
selfLink: /api/v1/namespaces/kube-system/pods/cluster-autoscaler-57bdc587bf-v66tp
uid: b99d5bfb-e46d-49eb-a986-eb2dfbdb8b65
spec:
containers:
- command:
- ./cluster-autoscaler
- --v=4
- --stderrthreshold=info
- --cloud-provider=aws
- --skip-nodes-with-local-storage=false
- --expander=priority
- --nodes=0:10:gamelift-gameservergroup-
- --nodes=0:10:gamelift-gameservergroup-agones-game-server-group-02
- --balance-similar-node-groups
- --skip-nodes-with-system-pods=false
env:
- name: AWS_REGION
value: eu-west-2
- name: AWS_ROLE_ARN
value: arn:aws:iam::974472462311:role/eksctl-agones-addon-iamserviceaccount-kube-s-Role1-1BU4LZK3QVXRZ
- name: AWS_WEB_IDENTITY_TOKEN_FILE
value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.18.3
imagePullPolicy: Always
name: cluster-autoscaler
resources:
limits:
cpu: 100m
memory: 300Mi
requests:
cpu: 100m
memory: 300Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/ssl/certs/ca-certificates.crt
name: ssl-certs
readOnly: true
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: cluster-autoscaler-token-2dvnx
readOnly: true
- mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
name: aws-iam-token
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
nodeName: ip-192-168-41-125.eu-west-2.compute.internal
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: cluster-autoscaler
serviceAccountName: cluster-autoscaler
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: aws-iam-token
projected:
defaultMode: 420
sources:
- serviceAccountToken:
audience: sts.amazonaws.com
expirationSeconds: 86400
path: token
- hostPath:
path: /etc/ssl/certs/ca-bundle.crt
type: ""
name: ssl-certs
- name: cluster-autoscaler-token-2dvnx
secret:
defaultMode: 420
secretName: cluster-autoscaler-token-2dvnx
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2021-05-11T12:52:55Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2021-05-11T12:52:58Z"
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2021-05-11T12:52:58Z"
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2021-05-11T12:52:55Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: docker://90cced865cbcc75209cd0c61c565a24d28e059ab2a83acebd882508fadc25c06
image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.18.3
imageID: docker-pullable://k8s.gcr.io/autoscaling/cluster-autoscaler@sha256:ad77e9587aff8cdaa90fb371cfff4458205ef63a4783734b5dd667872d1e91e1
lastState: {}
name: cluster-autoscaler
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2021-05-11T12:52:57Z"
hostIP: 192.168.41.125
phase: Running
podIP: 192.168.35.195
podIPs:
- ip: 192.168.35.195
qosClass: Guaranteed
startTime: "2021-05-11T12:52:55Z"
ec2-user:~/environment $
So it looks like you have 2 ASGs/game server groups:
--nodes=0:10:gamelift-gameservergroup-
--nodes=0:10:gamelift-gameservergroup-agones-game-server-group-02
It also looks like you're using the priority expander. I suggest deleting 1 of the 2 ASGs and using the least waste expander with auto discovery. Not sure why the guide has you create multiple game server groups. You should be able to get the capacity you need from a single ASG with diverse instance types that spans multiple AZs.
Thanks @jicowan I will look into deleting the groups. At this stage I am just testing agones and need to develop a wrapper and autoscaler - Ideally I'd like to run this for cheap or locally to avoid charges - following the guide with no mention of costs, I have accrued $200+ in charges! I just wanted to learn about the system.
Any advice?
How do I revert any chargeable infrastructure? I can't terminate the spot instances.
How long has the environment been running? The EKS control plane costs 10 cents an hour which comes to about $70/mo. You also pay for the provisioned compute. A majority of that should be Spot which costs 40% less (on average) than the on demand cost. FleetIQ doesn't offer as steep a discount as Spot. If you're primarily interested in testing Agones, you can try using Kubernetes in Docker (kind) assuming you have a machine powerful enough to run your game + Kubernetes. Definitely appreciate you trying the solution. Happy to continue working with you to get it working in your environment.
You need to delete the game server group to delete the ASGs. You can't delete the ASGs directly because they are managed by FleetIQ.
Think its been running since May 7th, I'm in the GameLift game server group area on the console and have no options to delete, might be permissions. Are the some commands I could run in the Cloud9 dev environment?
aws gamelift delete-game-server-group --game-server-group-name <your gsg name>
I think that'll do it.
Yeah that should do it, but I don't know how to force or shutdown the gameservers first?
ec2-user:~/environment $ aws gamelift delete-game-server-group --game-server-group-name agones-game-server-group-01
An error occurred (InvalidRequestException) when calling the DeleteGameServerGroup operation: Failed to SAFE_DELETE as there exists at least one UTILIZED game server
ec2-user:~/environment $ aws gamelift delete-game-server-group --game-server-group-name agones-game-server-group-02
An error occurred (InvalidRequestException) when calling the DeleteGameServerGroup operation: Failed to SAFE_DELETE as there exists at least one UTILIZED game server
ec2-user:~/environment $
No problem. Go to the ASG and set the min and desired count to 0 first.
Thanks - managed to set the ASG programatically with aws autoscale, but then I found the groups in the console anyway and have terminated the instances. Also found some ELBs. My services say I am still billed for EKS and some GameLift things but in those respective areas of the console I can't view anything to delete, even as the root user.
I've got a support case going on so they might be able to help me further.
I think I could try using agones locally for dev/test. But I was interested in using the AWS FlexMatch service too - I think that's cloud only?