Testkube-api downloads logs artifacts under incorrect S3 bucket path
KrPetrovDev opened this issue · 24 comments
Describe the bug
When I create and execute a simple curl-test from the testkube-dashboard UI, I cannot download the logs from the execution despite that I see that they are stored correctly in an S3 bucket in AWS.
Also this log artifact which has the following name 660fb2ff8358704f8a3f1d07 If I change It to be inside a directory named like the artifact which be resulting in the following path 660fb2ff8358704f8a3f1d07/660fb2ff8358704f8a3f1d07 testkube-api is able to install It correctly.
So the issue is that the way testkube-api has to upload the artifacts to S3 is different from the one when downloading them.
And as additional data I can see from testkube-api the following log line when trying to download the artifact:
{"level":"info","ts":1712306203.6748464,"caller":"minio/minio.go:159","msg":"listing files","bucket":"s3-bucket","bucketFolder":"660fb2ff8358704f8a3f1d07"}
To Reproduce
Steps to reproduce the behavior:
- Create kubernetes cluster
- Install testkube versions that I mentioned below
- Make testkube-api and testkube-dashboard reachable from outside of the cluster
- create the following yaml test to be executed from the testkube-dashboard:
apiVersion: tests.testkube.io/v3
kind: Test
metadata:
name: curl-test
namespace: app-project
labels:
executor: curl-executor
test-type: curl-test
spec:
type: curl/test
content:
type: string
data: "{\n "command": [\n "curl",\n"https://dummy.restapiexample.com/api/v1/employee/1",\n "-H",\n "Accept: application/json"\n ],\n "expected_status": "200"\n}"
status:
latestExecution:
id: 660fb2ff8358704f8a3f1d07
number: 1
startTime: 2024-04-05T08:14:55Z
endTime: 2024-04-05T08:14:58Z
status: passed
Expected behavior
I expected from testkube-api to know where is actually located the artifact and download It from there, which is this path 660fb2ff8358704f8a3f1d07 instead of this path 660fb2ff8358704f8a3f1d07/660fb2ff8358704f8a3f1d07
Version / Cluster
- Which testkube version?
- testkube helm chart version is 1.17.9
- testkube-dashboard helm chart version is 1.16.7
- testkube-api pod using the following docker image tag: 1.17.7
- What Kubernetes cluster?
- Running in a EKS AWS cluster
- What Kubernetes version?
- 1.27 kubernetes version
Additional context
I have followed the following Testkube documentation to configure Testkube to upload the artifacts in S3 instead of minIO:
Attaching the Slack thread of this issue for some additonal context: Slack thread
thank you @KrPetrovDev we will check it out
hey @KrPetrovDev can you please check it in CLI as well
kubectl testkube get artifact 660fb2ff8358704f8a3f1d07
What do you get?
@KrPetrovDev Can you post your (anonymized) Helm chart configuration for testkube-api storage and logs?
Contrary to some documentation I strongly suggest separating out artifacts and logs into separate S3 buckets. The server itself has some issues when these are combined, as far as I remember from what I figured out from our setup troubles and code analysis.
storage:
endpoint: "s3.amazonaws.com"
endpoint_port: ""
accessKeyId: ""
accessKey: ""
region: "eu-central-1"
bucket: "these-are-my-test-artifacts"
SSL: true
compressArtifacts: false
skipVerify: true
logs:
storage: "minio"
bucket: "these-are-my-test-logs"hey, @KrPetrovDev it should. the problem is, we removed it for OSS. So, it's for @TheBrunoLopes and @jmorante-ks to decide about
Hello @vsukhin, I understand, so Is there any workaround for the open-source solution that allows me download the log artifacts from the UI meanwhile?,
like using minIO instead or using a PVC attached to testkube-api pod to persist the artifact objects
does cli works in this case?
Do you mean kubectl testkube?
If that's the case, kubectl does not recognise testkube command.
Here It is the log that It gives me when I try to use It:
error: unknown command "testkube" for "kubectl"
you will need to install cli https://docs.testkube.io/articles/install-cli
okey, once that I have It installed and I try to install the logs artifact it retrieves me the log below with this command kubectl testkube download --namespace=app-project artifacts 660fb2ff8358704f8a3f1d07:
Log from the command:
`Context: (1.17.14) Namespace: app-project
Getting artifacts count = 1
downloading file: (error: an error on the server ("unknown") has prevented the request from succeeding (get services testkube-api-server:8088))
`
Log from the testkube-api-server pod:
{“level”:“error”,“ts”:1713441191.9673371,“caller”:“server/httpserver.go:76”,“msg”:“failed to get artifact 660fb2ff8358704f8a3f1d07 for execution 660fb2ff8358704f8a3f1d07: could not download file: minio DownloadFile err │ │ or: minio Download File Stat error: The specified key does not exist., error from getting files from former bucket per execution: <nil>“,”status”:500,“stacktrace”:“github.com/kubeshop/testkube/pkg/server.(*HTTPServer). │ │ Error\n\t/home/runner/work/testkube/testkube/pkg/server/httpserver.go:76\ngithub.com/kubeshop/testkube/internal/app/api/v1.(*TestkubeAPI).InitRoutes.(*TestkubeAPI).GetArtifactHandler.func27\n\t/home/runner/work/testkub │ │ e/testkube/internal/app/api/v1/executions.go:462\ngithub.com/gofiber/fiber/v2.(*App).next\n\t/home/runner/go/pkg/mod/github.com/gofiber/fiber/v2@v2.51.0/router.go:145\ngithub.com/gofiber/fiber/v2.(*Ctx).Next\n\t/home/r │ │ unner/go/pkg/mod/github.com/gofiber/fiber/v2@v2.51.0/ctx.go:1030\ngithub.com/kubeshop/testkube/internal/app/api/v1.(*TestkubeAPI).InitRoutes.(*TestkubeAPI).AuthHandler.func2\n\t/home/runner/work/testkube/testkube/inter │ │ nal/app/api/v1/handlers.go:47\ngithub.com/gofiber/fiber/v2.(*Ctx).Next\n\t/home/runner/go/pkg/mod/github.com/gofiber/fiber/v2@v2.51.0/ctx.go:1027\ngithub.com/gofiber/fiber/v2/middleware/cors.New.func1\n\t/home/runner/g │ │ o/pkg/mod/github.com/gofiber/fiber/v2@v2.51.0/middleware/cors/cors.go:165\ngithub.com/gofiber/fiber/v2.(*App).next\n\t/home/runner/go/pkg/mod/github.com/gofiber/fiber/v2@v2.51.0/router.go:145\ngithub.com/gofiber/fiber/ │ │ v2.(*Ctx).Next\n\t/home/runner/go/pkg/mod/github.com/gofiber/fiber/v2@v2.51.0/ctx.go:1030\ngithub.com/gofiber/fiber/v2/middleware/pprof.New.func1\n\t/home/runner/go/pkg/mod/github.com/gofiber/fiber/v2@v2.51.0/middlewar │ │ e/pprof/pprof.go:46\ngithub.com/gofiber/fiber/v2.(*Ctx).Next\n\t/home/runner/go/pkg/mod/github.com/gofiber/fiber/v2@v2.51.0/ctx.go:1027\ngithub.com/kubeshop/testkube/pkg/server.(*HTTPServer).Init.func1\n\t/home/runner/ │ │ work/testkube/testkube/pkg/server/httpserver.go:46\ngithub.com/gofiber/fiber/v2.(*App).next\n\t/home/runner/go/pkg/mod/github.com/gofiber/fiber/v2@v2.51.0/router.go:145\ngithub.com/gofiber/fiber/v2.(*App).handler\n\t/h │ │ ome/runner/go/pkg/mod/github.com/gofiber/fiber/v2@v2.51.0/router.go:172\ngithub.com/valyala/fasthttp.(*Server).serveConn\n\t/home/runner/go/pkg/mod/github.com/valyala/fasthttp@v1.50.0/server.go:2359\ngithub.com/valyala │ │ /fasthttp.(*workerPool).workerFunc\n\t/home/runner/go/pkg/mod/github.com/valyala/fasthttp@v1.50.0/workerpool.go:224\ngithub.com/valyala/fasthttp.(*workerPool).getCh.func1\n\t/home/runner/go/pkg/mod/github.com/valyala/f │ │ asthttp@v1.50.0/workerpool.go:196"}
Also when I try to perform the same change that I explained before with testkube-dashboard It allows me to download It,
Which is introducing the file into a folder with the same name as the file, which will result in the following path inside the S3 bucket as an example:
660fb2ff8358704f8a3f1d07/660fb2ff8358704f8a3f1d07
@vsukhin If I remember correctly there is a bug thay happens because the filenames of logs in the S3 bucket clash with the folder names used to store artifacts.
@KrPetrovDev ok, but what is listed by kubectl testkube get artifacts 660fb2ff8358704f8a3f1d07 --namespace=app-project
@frederikb could be a reason. @KrPetrovDev do you have separate folders for atifacts and logs in your S3 bucket?
@vsukhin no, I don't have
Sounds like, we definitely need 2 separate buckets for artifacts and logs
Because of minio/minio#17574
Real folders for artifacts and virtual folders for logs
Does usage of different buckets help/
so closing with explanation above. feel free to reopen, if it's not enough
@vsukhin but the helm chart only allows me to specify one bucket, how do you want me to specify these two buckets and configure each one for the different kind of objects(artifacts and logs)?
okay, I see, will try this option
@KrPetrovDev I posted an example configuration snippet above, which you could check out.