Locust Report not uploading

Question

Locust Report not uploading

Closed this issue 3 years ago · 34 comments

@diegomarangoni I have installed 1.1.11 as you mentioned in my previous issue. After spinning up the pods, when I try to upload a sample file, I am getting below error.

curl -X PUT -T kangal.yaml -L http://localhost:8081/load-test/loadtest-orderly-armadillo/report -v
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8081 (#0)
> PUT /load-test/loadtest-orderly-armadillo/report HTTP/1.1
> Host: localhost:8081
> User-Agent: curl/7.58.0
> Accept: */*
> Content-Length: 2637
> Expect: 100-continue
>
* Done waiting for 100-continue
* We are completely uploaded and fine
< HTTP/1.1 500 Internal Server Error
< Content-Type: application/json; charset=utf-8
< Vary: Origin
< Date: Tue, 20 Apr 2021 16:29:12 GMT
< Content-Length: 75
< Connection: close
<
{"error":"Presigned URLs cannot be generated with anonymous credentials."}
* Closing connection 0

Answer 1 · 2021-04-20T16:33:55.000Z

Here is the log from the locust pod.

[2021-04-20 16:30:11,159] loadtest-orderly-armadillo-master-jljkq/INFO/locust.main: Time limit reached. Stopping Locust.
[2021-04-20 16:30:11,162] loadtest-orderly-armadillo-master-jljkq/INFO/locust.runners: Client 'loadtest-orderly-armadillo-worker-4h5j7_c60cbc9d95fa4703a4b876daaf4de31b' quit. Currently 0 clients connected.
[2021-04-20 16:30:11,163] loadtest-orderly-armadillo-master-jljkq/INFO/locust.runners: The last worker quit, stopping test.
 Name                                                          # reqs      # fails  |     Avg     Min     Max  Median  |   req/s failures/s
--------------------------------------------------------------------------------------------------------------------------------------------
 GET /                                                             34     0(0.00%)  |      19      16      58      18  |    0.50    0.00
--------------------------------------------------------------------------------------------------------------------------------------------
 Aggregated                                                        34     0(0.00%)  |      19      16      58      18  |    0.50    0.00

[2021-04-20 16:30:11,663] loadtest-orderly-armadillo-master-jljkq/INFO/locust.main: Running teardowns...
Adding /tmp/report_failures.csv...
Adding /tmp/report_stats.csv...
Adding /tmp/report_stats_history.csv...
[2021-04-20 16:30:12,218] loadtest-orderly-armadillo-master-jljkq/ERROR/root: Uncaught exception in event handler:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 159, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 61, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/local/lib/python3.8/site-packages/gevent/_socketcommon.py", line 230, in getaddrinfo
    addrlist = get_hub().resolver.getaddrinfo(host, port, family, type, proto, flags)
  File "/usr/local/lib/python3.8/site-packages/gevent/resolver/thread.py", line 63, in getaddrinfo
    return self.pool.apply(_socket.getaddrinfo, args, kwargs)
  File "/usr/local/lib/python3.8/site-packages/gevent/pool.py", line 161, in apply
    return self.spawn(func, *args, **kwds).get()
  File "src/gevent/event.py", line 305, in gevent._gevent_cevent.AsyncResult.get
  File "src/gevent/event.py", line 335, in gevent._gevent_cevent.AsyncResult.get
  File "src/gevent/event.py", line 323, in gevent._gevent_cevent.AsyncResult.get
  File "src/gevent/event.py", line 303, in gevent._gevent_cevent.AsyncResult._raise_exception
  File "/usr/local/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.8/site-packages/gevent/threadpool.py", line 142, in __run_task
    thread_result.set(func(*args, **kwargs))
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 309, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 171, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f1310681eb0>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='kangal-proxy.local', port=443): Max retries exceeded with url: /load-test/loadtest-orderly-armadillo/report (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f1310681eb0>: Failed to establish a new connection: [Errno -2] Name or service not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/locust/event.py", line 40, in fire
    handler(**kwargs)
  File "/data/locustfile.py", line 33, in hook_quit
    r = requests.put(presigned_url, data=open(report, 'rb'), headers=request_headers)
  File "/usr/local/lib/python3.8/site-packages/requests/api.py", line 134, in put
    return request('put', url, data=data, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='kangal-proxy.local', port=443): Max retries exceeded with url: /load-test/loadtest-orderly-armadillo/report (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f1310681eb0>: Failed to establish a new connection: [Errno -2] Name or service not known'))

[2021-04-20 16:30:12,219] loadtest-orderly-armadillo-master-jljkq/INFO/locust.main: Shutting down (exit code 2), bye.
[2021-04-20 16:30:12,219] loadtest-orderly-armadillo-master-jljkq/INFO/locust.main: Cleaning up runner...

Answer 2 · 2021-04-20T17:00:36.000Z

hey @QAInsights, looks like we have multiple errors here.

Let's start first with Locust pod error:
what I can see is that locust is trying to establish a connection using the address https://kangal-proxy.local.

this value comes from variable controller.env.KANGAL_PROXY_URL, this address must point to kangal-proxy-k8s Service, so assuming you installed to default namespace, it should be something similar to this:
http://kangal-proxy-k8s.default.svc.cluster.local, note that I am using HTTP (port 80) and not HTTPS (port 443).

Now, regarding the curl request, I assuming you doing a kubectl port-forward to Kangal proxy pod and are trying to manually upload a report, if that is the case, then:

The error Presigned URLs cannot be generated with anonymous credentials. means the AWS credentials are not set, can you check if the Pod is correctly using the configMap (or secret if you using the latest chart version)?

Answer 3 · 2021-04-20T17:03:03.000Z

Thanks @diegomarangoni I see github action released helm version. I just tried with the helm release with the aws credentials. Pods are not getting created, it is throwing CreateContainerConfigError for all the pods.

Answer 4 · 2021-04-20T17:03:49.000Z

what command did you use to install?

Answer 5 · 2021-04-20T17:06:55.000Z

You have to specify secrets.AWS_* on either, command-line or values file, as described here:
https://github.com/hellofresh/kangal/tree/master/charts/kangal

Answer 6 · 2021-04-20T17:09:53.000Z

FYI, I will put this on our backlog to discuss internally some ideas to improve and make it easier the installation process.

Answer 7 · 2021-04-20T17:37:51.000Z

I used

helm install   --set environment=dev   --set secrets.AWS_ACCESS_KEY_ID="XXXXXXXXXXXXXX"   --set secrets.AWS_SECRET_ACCESS_KEY="XXXXXXXXXXX"   kangal kangal/kangal

Answer 8 · 2021-04-20T17:41:28.000Z

Seems that you are installing from the remote repository, make sure you run helm repo update to fetch the latest update

Answer 9 · 2021-04-20T17:42:18.000Z

I deleted the helm installation and tried the above command. Pods are getting created. I executed a simple test in locust. Here is the error.

[2021-04-20 17:40:14,547] loadtest-lumbering-echidna-master-ckqrz/INFO/locust.main: Running teardowns...
Adding /tmp/report_failures.csv...
Adding /tmp/report_stats.csv...
Adding /tmp/report_stats_history.csv...
[2021-04-20 17:40:15,127] loadtest-lumbering-echidna-master-ckqrz/ERROR/root: Uncaught exception in event handler:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 159, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 61, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/local/lib/python3.8/site-packages/gevent/_socketcommon.py", line 230, in getaddrinfo
    addrlist = get_hub().resolver.getaddrinfo(host, port, family, type, proto, flags)
  File "/usr/local/lib/python3.8/site-packages/gevent/resolver/thread.py", line 63, in getaddrinfo
    return self.pool.apply(_socket.getaddrinfo, args, kwargs)
  File "/usr/local/lib/python3.8/site-packages/gevent/pool.py", line 161, in apply
    return self.spawn(func, *args, **kwds).get()
  File "src/gevent/event.py", line 305, in gevent._gevent_cevent.AsyncResult.get
  File "src/gevent/event.py", line 335, in gevent._gevent_cevent.AsyncResult.get
  File "src/gevent/event.py", line 323, in gevent._gevent_cevent.AsyncResult.get
  File "src/gevent/event.py", line 303, in gevent._gevent_cevent.AsyncResult._raise_exception
  File "/usr/local/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.8/site-packages/gevent/threadpool.py", line 142, in __run_task
    thread_result.set(func(*args, **kwargs))
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 309, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 171, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7ff1e33d8d90>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='kangal-proxy.local', port=443): Max retries exceeded with url: /load-test/loadtest-lumbering-echidna/report (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff1e33d8d90>: Failed to establish a new connection: [Errno -2] Name or service not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/locust/event.py", line 40, in fire
    handler(**kwargs)
  File "/data/locustfile.py", line 33, in hook_quit
    r = requests.put(presigned_url, data=open(report, 'rb'), headers=request_headers)
  File "/usr/local/lib/python3.8/site-packages/requests/api.py", line 134, in put
    return request('put', url, data=data, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='kangal-proxy.local', port=443): Max retries exceeded with url: /load-test/loadtest-lumbering-echidna/report (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff1e33d8d90>: Failed to establish a new connection: [Errno -2] Name or service not known'))

[2021-04-20 17:40:15,127] loadtest-lumbering-echidna-master-ckqrz/INFO/locust.main: Shutting down (exit code 2), bye.
[2021-04-20 17:40:15,128] loadtest-lumbering-echidna-master-ckqrz/INFO/locust.main: Cleaning up runner...

Answer 10 · 2021-04-20T17:42:52.000Z

I am open to share my screen, in case if you want to take a deep-dive. @diegomarangoni

Answer 11 · 2021-04-20T17:46:55.000Z

URL stills the same:

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='kangal-proxy.local', port=443):

You have to define controller.env.KANGAL_PROXY_URL, like this:

helm install  \
  --set environment=dev \
  --set secrets.AWS_ACCESS_KEY_ID="XXXXXXXXXXXXXX" \
  --set secrets.AWS_SECRET_ACCESS_KEY="XXXXXXXXXXX" \
  --set controller.env.KANGAL_PROXY_URL="http://kangal-proxy-k8s.default.svc.cluster.local" \
  kangal kangal/kangal

Answer 12 · 2021-04-20T17:57:42.000Z

Finally no error :) But the report URL is throwing Access Denied. Reports are not there in S3 bucket.

curl 'http://127.0.0.1:8081/load-test/loadtest-unsung-boxer/report/'
Access Denied.

Answer 13 · 2021-04-20T17:59:59.000Z

Ok, seems we are doing some progress :)
Do the credentials have write access to the bucket?

Can you check Locust pod logs?

Answer 14 · 2021-04-20T18:01:48.000Z

Here is the pod log.

[2021-04-20 17:54:47,354] loadtest-unsung-boxer-master-5xpgp/INFO/locust.main: Time limit reached. Stopping Locust.
[2021-04-20 17:54:47,358] loadtest-unsung-boxer-master-5xpgp/INFO/locust.runners: Client 'loadtest-unsung-boxer-worker-gs69h_f0be8065575549c39ab26351b5324fa1' quit. Currently 0 clients connected.
[2021-04-20 17:54:47,358] loadtest-unsung-boxer-master-5xpgp/INFO/locust.runners: The last worker quit, stopping test.

Answer 15 · 2021-04-20T18:03:12.000Z

Can you send the part related to report uploading?

Answer 16 · 2021-04-20T18:04:19.000Z

I dont see any upload log info.

[2021-04-20 17:54:47,857] loadtest-unsung-boxer-master-5xpgp/INFO/locust.main: Running teardowns...
Adding /tmp/report_failures.csv...
Adding /tmp/report_stats.csv...
Adding /tmp/report_stats_history.csv...
[2021-04-20 17:54:48,490] loadtest-unsung-boxer-master-5xpgp/INFO/locust.main: Shutting down (exit code 0), bye.
[2021-04-20 17:54:48,491] loadtest-unsung-boxer-master-5xpgp/INFO/locust.main: Cleaning up runner...

Answer 17 · 2021-04-20T18:05:32.000Z

Ok, Locust seems fine, let's see Kangal Proxy logs

Answer 18 · 2021-04-20T18:08:24.000Z

curl 'http://127.0.0.1:8081/load-test/loadtest-unsung-boxer/report/'
Access Denied.

Can you double-check if the user you are using have write permission to this bucket?

Answer 19 · 2021-04-20T18:15:26.000Z

Could you please share the sample bucket policy? I have enabled public access as of now.

Answer 20 · 2021-04-20T18:18:37.000Z

Got some error in the proxy.

{"level":"error","ts":"2021-04-20T17:57:03.728Z","caller":"kubernetes/client.go:107","msg":"Error on retrieving info for loadtest","type":"kangal","loadtest":"report","error":"loadtests.kangal.hellofresh.com \"report\" not found","stacktrace":"github.com/hellofresh/kangal/pkg/kubernetes.(*Client).GetLoadTest\n\t/home/runner/work/kangal/kangal/pkg/kubernetes/client.go:107\ngithub.com/hellofresh/kangal/pkg/proxy.(*Proxy).Get\n\t/home/runner/work/kangal/kangal/pkg/proxy/proxy.go:238\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.15.11/x64/src/net/http/server.go:2042\ngo.opencensus.io/plugin/ochttp.WithRouteTag.func1\n\t/home/runner/go/pkg/mod/go.opencensus.io@v0.22.4/plugin/ochttp/route.go:40\ngo.opencensus.io/plugin/ochttp.taggedHandlerFunc.ServeHTTP\n\t/home/runner/go/pkg/mod/go.opencensus.io@v0.22.4/plugin/ochttp/route.go:51\ngithub.com/go-chi/chi.(*Mux).routeHTTP\n\t/home/runner/go/pkg/mod/github.com/go-chi/chi@v4.1.2+incompatible/mux.go:431\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.15.11/x64/src/net/http/server.go:2042\ngithub.com/rs/cors.(*Cors).Handler.func1\n\t/home/runner/go/pkg/mod/github.com/rs/cors@v1.7.0/cors.go:219\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.15.11/x64/src/net/http/server.go:2042\ngithub.com/go-chi/render.SetContentType.func1.1\n\t/home/runner/go/pkg/mod/github.com/go-chi/render@v1.0.1/content_type.go:52\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.15.11/x64/src/net/http/server.go:2042\ngithub.com/hellofresh/kangal/pkg/core/middleware.Recovery.func1\n\t/home/runner/work/kangal/kangal/pkg/core/middleware/recovery.go:33\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.15.11/x64/src/net/http/server.go:2042\ngithub.com/felixge/httpsnoop.CaptureMetrics.func1\n\t/home/runner/go/pkg/mod/github.com/felixge/httpsnoop@v1.0.1/capture_metrics.go:30\ngithub.com/felixge/httpsnoop.CaptureMetricsFn\n\t/home/runner/go/pkg/mod/github.com/felixge/httpsnoop@v1.0.1/capture_metrics.go:81\ngithub.com/felixge/httpsnoop.CaptureMetrics\n\t/home/runner/go/pkg/mod/github.com/felixge/httpsnoop@v1.0.1/capture_metrics.go:29\ngithub.com/hellofresh/kangal/pkg/core/middleware.(*RequestLogger).Handler.func1\n\t/home/runner/work/kangal/kangal/pkg/core/middleware/request_logger.go:25\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.15.11/x64/src/net/http/server.go:2042\ngithub.com/hellofresh/kangal/pkg/core/middleware.(*Logger).Handler.func1\n\t/home/runner/work/kangal/kangal/pkg/core/middleware/logger.go:28\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.15.11/x64/src/net/http/server.go:2042\ngithub.com/go-chi/chi/middleware.RealIP.func1\n\t/home/runner/go/pkg/mod/github.com/go-chi/chi@v4.1.2+incompatible/middleware/realip.go:34\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.15.11/x64/src/net/http/server.go:2042\ngithub.com/go-chi/chi/middleware.RequestID.func1\n\t/home/runner/go/pkg/mod/github.com/go-chi/chi@v4.1.2+incompatible/middleware/request_id.go:76\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.15.11/x64/src/net/http/server.go:2042\ngithub.com/go-chi/chi.(*Mux).ServeHTTP\n\t/home/runner/go/pkg/mod/github.com/go-chi/chi@v4.1.2+incompatible/mux.go:86\ngo.opencensus.io/plugin/ochttp.(*Handler).ServeHTTP\n\t/home/runner/go/pkg/mod/go.opencensus.io@v0.22.4/plugin/ochttp/server.go:92\nnet/http.serverHandler.ServeHTTP\n\t/opt/hostedtoolcache/go/1.15.11/x64/src/net/http/server.go:2843\nnet/http.(*conn).serve\n\t/opt/hostedtoolcache/go/1.15.11/x64/src/net/http/server.go:1925"}

Answer 21 · 2021-04-20T18:31:05.000Z

This logs have to relation with the curl 'http://127.0.0.1:8081/load-test/loadtest-unsung-boxer/report/' request

Try using the AWS builtin policy arn:aws:iam::aws:policy/AmazonS3FullAccess, if it works you can change it later

Answer 22 · 2021-04-20T18:35:14.000Z

Sure, so how can I set the config like bucket name and region after the pods are created. Do I have edit the config map?

Answer 23 · 2021-04-20T18:36:22.000Z

Yes, you can edit the config map by using:
kubectl -n default edit configmap kangal, where default is your namespace

Answer 24 · 2021-04-20T19:06:49.000Z

I do have configured the bucket with public access. Still access is getting denied.

Answer 25 · 2021-04-21T03:00:16.000Z

curl 'http://localhost:8081/load-test/loadtest-sappy-emu/report/' -v
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8081 (#0)
> GET /load-test/loadtest-sappy-emu/report/ HTTP/1.1
> Host: localhost:8081
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 500 Internal Server Error
< Content-Type: text/plain; charset=utf-8
< Vary: Origin
< X-Content-Type-Options: nosniff
< Date: Wed, 21 Apr 2021 02:59:41 GMT
< Content-Length: 15
<
Access Denied.

Answer 26 · 2021-04-21T08:43:14.000Z

AWS_DEFAULT_REGION is set to us-east-2 and AWS_ENDPOINT_URL is pointing to us-east-1.
Not sure if this is the cause, but seems related with AWS credentials

Answer 27 · 2021-04-21T14:35:28.000Z

Yes, I have changed to us-east-2 in the URL, still the access is denied.

Answer 28 · 2021-04-21T14:49:16.000Z

This is my S3 bucket policy

{
    "Version": "2012-10-17",
    "Id": "Policy1618970981834",
    "Statement": [
        {
            "Sid": "Stmt1618970979985",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::kangal-report"
        }
    ]
}

Answer 29 · 2021-04-21T15:19:15.000Z

hey @QAInsights, I just did a test here and it's working fine I can see the file on the bucket I've just created.
Here is the policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "*"
        }
    ]
}

The steps I did:

Go to IAM
Create new User with programmatically access
Attach policy AmazonS3FullAccess to this user
Generate new key
Copy Key ID and Secret to values.yaml file
Helm install

Answer 30 · 2021-04-21T16:01:36.000Z

I have followed the exact steps in AWS. Created a new user and added S3 full access. Configured the key and secret in the secrets. Access is denied even for GET report.

 curl GET 'http://127.0.0.1:8081/load-test/loadtest-altered-fox/report/'
curl: (6) Could not resolve host: GET
Access Denied.

@diegomarangoni

Answer 31 · 2021-04-21T16:24:31.000Z

I can't reproduce this issue. Steps I did:

Install with command:

helm install \
    --set environment=dev \
    --set configMap.AWS_DEFAULT_REGION=eu-west-1 \
    --set configMap.AWS_ENDPOINT_URL=s3.eu-west-1.amazonaws.com \
    --set configMap.AWS_BUCKET_NAME=test-kangal \
    --set secrets.AWS_ACCESS_KEY_ID="XXXXXXXXXXXXXXXXXXXX" \
    --set secrets.AWS_SECRET_ACCESS_KEY="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" \
    --set controller.env.KANGAL_PROXY_URL="http://kangal-proxy-k8s.default.svc.cluster.local" \
    kangal kangal/kangal

Port-forward to have access to Proxy:

kubectl port-forward svc/kangal-proxy-k8s 8080:80

Run a simple Locust load test:

http --form POST http://localhost:8080/load-test \
  distributedPods=1 \
  testFile@kangal-loadtest.py \
  type=Locust \
  duration=1m

HTTP/1.1 201 Created
Content-Length: 146
Content-Type: application/json; charset=utf-8
Date: Wed, 21 Apr 2021 16:17:43 GMT
Vary: Origin

{
    "distributedPods": 1,
    "hasEnvVars": false,
    "hasTestData": false,
    "loadtestName": "loadtest-hipster-koala",
    "phase": "creating",
    "tags": {},
    "type": "Locust"
}

Report appears on S3 bucket:

Answer 32 · 2021-04-21T20:05:15.000Z

Many thanks. I used configmap instead of configMap. It is working fine :)

Answer 33 · 2021-04-22T03:57:52.000Z

@diegomarangoni Please check my blog post :) https://qainsights.com/performance-testing-in-kubernetes-using-kangal/

Answer 34 · 2021-04-22T06:59:05.000Z

Glad it works @QAInsights!

@diegomarangoni Please check my blog post :) https://qainsights.com/performance-testing-in-kubernetes-using-kangal/

Thanks for sharing and for contributing to our community :)