Give 401 auth error after using kubectl testkube api

[TWpower]

Describe the bug
When using kubectl testkube api, the call works fine initially, but after about an hour, it returns the following error. The error occurs at the end after using the --watch command. Could you help me identify the problem? My GCP authentication is set to expire after 4 hours. Where and why is the authorization fail?

To Reproduce
Steps to reproduce the behavior:

1. Run "kubectl testkube run test $test_name --namespace $namespace --watch"
2. See error

Request URI:	http://10.7.106.111/infra/target-sources/1457/assume-roles/?isReadOnly=false
07:23:16.819 [main] INFO  c.s.b.d.i.i.CheckServiceCodeExistTest - Cleanup of assume role done
07:23:16.820 [main] INFO  c.s.b.d.i.i.CheckServiceCodeExistTest - cleaning test environment done
[INFO] Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3649 s -- in com.samsung.bigdata.dip.integrationtest.pipeline.meta.SchemaManagerLoadTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 19, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:01 h
[INFO] Finished at: 2024-09-26T07:23:16Z
[INFO] ------------------------------------------------------------------------

get test execution details (error: api/GET-testkube.Execution returned error: api server response: '{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

'
error: Unauthorized)

Error: Process completed with exit code 1.

Expected behavior

• It should not give me a authorization error.

Version / Cluster

• Which testkube version? 1.16.39
• What Kubernetes cluster? GKE
• What Kubernetes version?

Client Version: v1.30.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.3-gke.1969001

[vsukhin]

hey, @TWpower
Looks like an interesting issue.

I saw some requests about mismatching between token expiration and policy settings
https://www.googlecloudcommunity.com/gc/Cloud-Hub/gcloud-token-is-expiring-every-1hour/td-p/733736

But might be, it was just near the end of token expiration tine, when you started the test, so it's expired during test executiom? So,

[TWpower]

@vsukhin
Hello
Thanks for your reply.
I checked that gcp auth is fine.
I set it to 14400s(seconds).

And also checked that only testkube api give me a 401 error.
I tried after 1 hour with command gcloud compute instances list but it works fine.

[TWpower]

• normal result

Run gcloud compute instances list
WARNING: Some requests did not succeed.
 - Required 'compute.instances.list' permission for 'projects/sec-bdp-dev'

Listed 0 items.

• error after expired

Run gcloud compute instances list
ERROR: (gcloud.compute.instances.list) There was a problem refreshing your current auth tokens: ('Unable to retrieve Identity Pool subject token', '{"$id":"1","innerException":null,"message":"Can\'t issue ID_TOKEN for job in \'Completed\' state.","typeName":"GitHub.Actions.Runtime.WebApi.CannotGenerateIdTokenException, GitHub.Actions.Runtime.WebApi, Version=14.0.0.0, Culture=neutral, PublicKeyToken=null","typeKey":"CannotGenerateIdTokenException","errorCode":0,"eventId":3000}')
Please run:

  $ gcloud auth login

to obtain new credentials.

If you have already logged in with a different account, run:

  $ gcloud config set account ACCOUNT

to select an already authenticated account to use.
Error: Process completed with exit code 1.

• testkube api result whenever after 1 hour even though expiration is set to more that an hour

Run kubectl testkube get executions --namespace integration-test --verbose --test integration-all-task-test 

Context: kubeconfig (1.16.39)   Namespace: integration-test
-----------------------------------------------------------
Getting executions for test: integration-all-task-test (error: api/GET-testkube.ExecutionsResult returned error: api server response: '{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

'
error: Unauthorized)

[vsukhin]

hey, @TWpower
Testkube doesn't use its own auth to reach k8s cluster, it's fully based on go k8s client. I' never tried expired gcp auth tokens, might be our devops team @ypoplavs @dejanzele has better ideas what might be wrong
Just to check, if your token has 24 hrs or 48 hrs ttl, it sitll fails?

Do you use any special auth plugins? Might be they are ignored by embedded client?

[TWpower]

@vsukhin Hello
I found that there was an issue in gke credential of our cluster (which only valid for 1 hour)
That was the reason why it not works!

Thanks for your help! I think I can handle it from now!

[vsukhin]

welcome and hppy testkubing!