AOSS gw-helper-deny

[adrianluisgonzalez]

Thanks for this helpful tool. I have used it with Amazon OpenSearch clusters for a while now. I am trying to move to OpenSearch Serverless, but having issues with aws-es-proxy when there is a payload to be signed.

When I send request without a payload, everything works fine. For example curl localhost:9200/_cat/indices or curl -X POST localhost:9200/my-index/_search.

As soon as I send any request with a payload, I get a 403 with response header X-Aoss-Response-Hint: X01:gw-helper-deny

Here are the logs using v1.5:

aws-es-proxy --endpoint https://xxxxxxxxxx.eu-west-1.aoss.amazonaws.com -debug -verbose                                                                                                                                                   

DEBU[2024-10-01 13:08:42] Provided endpoint is a valid AWS Elasticsearch endpoint 
DEBU[2024-10-01 13:08:42] AWS Region eu-west-1                         
INFO[2024-10-01 13:08:42] Listening on 127.0.0.1:9200...               
INFO[2024-10-01 13:08:55] Using default credentials                    
INFO[2024-10-01 13:08:55] Generated fresh AWS Credentials object       
2024/01/10 13:08:56  -> GET; 127.0.0.1:65405; /_cat/indices; ; 200; 0.672s
2024/01/10 13:09:09  -> GET; 127.0.0.1:65427; /my-index/_search; ; 200; 0.114s
ERRO[2024-10-01 13:09:19] Received 403 from AWSAuth, invalidating credentials for retrial 
DEBU[2024-10-01 13:09:19] Received Status code from AWS: 403           
DEBU[2024-10-01 13:09:19] Received headers from AWS: map[Content-Length:[121] Content-Type:[application/json] Date:[Wed, 10 Jan 2024 12:09:19 GMT] Server:[aoss-amazon] X-Aoss-Response-Hint:[X01:gw-helper-deny] X-Request-Id:[48b0cd33-ea09-9cde-a2fe-e9d36f11ddb8]] 
DEBU[2024-10-01 13:09:19] Received body from AWS: {"status":403,"request-id":"48b0cd33-ea09-9cde-a2fe-e9d36f11ddb8","error":{"reason":"403 Forbidden","type":"Forbidden"}} 
2024/01/10 13:09:19  -> POST; 127.0.0.1:65442; /my-index/_search; {}; 403; 0.155s

The serverless data access policy has full access for my IAM credentials and the fact the search without a payload succeeds makes me think this is not permissions related. I don't see anything in CloudTrail.

Has anyone used this successfully with AOSS? Any suggestions would be greatly appreciated.

[adrianluisgonzalez]

I found the issue. AOSS needs the header X-Amz-Content-Sha256

https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-clients.html#serverless-signing

You can see the following in opensearch-go:

	contentSha256Hash := emptyBodySHA256

	if req.Body != nil {
		b, err := io.ReadAll(req.Body)
		if err != nil {
			return fmt.Errorf("failed to read request body: %w", err)
		}

		body = bytes.NewReader(b)

		hash, err := hexEncodedSha256(b)
		if err != nil {
			return fmt.Errorf("failed to calculate hash of request body: %w", err)
		}

		contentSha256Hash = hash
	}
	// Add the "X-Amz-Content-Sha256" header as required by Amazon OpenSearch Serverless.
	req.Header.Set("X-Amz-Content-Sha256", contentSha256Hash)

https://github.com/opensearch-project/opensearch-go/blob/main/signer/aws/aws.go#L95