How should the CloudWatch agent be configured with credentials when it sends logs to CloudWatch in other regions?

My instance is in us-east-1. I want to send logs to CloudWatch in us-gov-west-1. How should I configure the credentials?

I followed the documentation to create an AmazonCloudWatchAgent configuration file for the CloudWatch agent

sudo aws configure --profile AmazonCloudWatchAgent

Configured credentials and regions related to the us-gov-west-1 region

sudo aws configure --profile AmazonCloudWatchAgent list

`Name Value Type Location
  ----                    -----             ----    --------
profile AmazonCloudWatchAgent manual --profile
access_key ****************s77H shared-credentials-file
secret_key ****************7K7k shared-credentials-file
region            us-gov-west-1      config-file    ~/.aws/config`

Why does it still report an error when running?

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m onPremise -s -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

`****** processing amazon-cloudwatch-agent ******

Got Home directory: /root I! Set home dir Linux: /root I! SDKRegionWithCredsMap region: us-gov-west-1 Successfully fetched the config and saved in /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_amazon-cloudwatch-agent.json.tmp
Start configuration validation...
2024/01/09 08:05:15 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_amazon-cloudwatch-agent.json.tmp ...
2024/01/09 08:05:15 unable to scan config dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d with error: unable to parse json, error: unexpected end of JSON input
2024/01/09 08:05:15 No json config files found, use the default one
2024/01/09 08:05:15 I! Valid Json input schema.
2024/01/09 08:05:15 E! Credentials path is not set while runasuser is not root
2024/01/09 08:05:15 E! Credentials path is not set while runasuser is not root
2024/01/09 08:05:15 Configuration validation first phase failed. Agent version: 1.0. Verify the JSON input is only using features supported by this version.`

amazon-cloudwatch-agent.json and common-config.toml are copied and edited manually from the aws document example. Do I need to modify anything else?

Currently the CloudWatch agent can be started, but the security token is wrong and the logs cannot be uploaded to the gov zone, but there is a conflict between the IAM role of the EC2 attachment and the configured credentials.

/root/.aws/credentials does not make any configuration error content

2024-01-10T07:16:47Z E! [outputs.cloudwatchlogs] Aws error received when sending logs to ubuntu_ec2_status/CICDMirror_Test: UnrecognizedClientException: The security token included in the request is invalid
2024-01-10T07:16:47Z W! [outputs.cloudwatchlogs] Retried 15 time, going to sleep 33.853227985s before retrying.
2024-01-10T07:16:51Z E! cloudwatch: code: InvalidClientTokenId, message: The security token included in the request is invalid, original error: <nil>
2024-01-10T07:16:51Z W! cloudwatch: 27 retries, going to sleep 44077 ms before retrying.
2024-01-10T07:16:56Z E! cloudwatch: WriteToCloudWatch failure, err:  InvalidClientTokenId: The security token included in the request is invalid
	status code: 403, request id: fb180a3f-c826-4a5d-aae3-8a4ae99ca77d

Configure the error content of /root/.aws/credentials

2024-01-10T07:33:00Z W! Unused shared config file(s) found: [/root/.aws/credentials]. If you would like to use them, please update your common-config.toml.
2024-01-10T07:33:00Z W! Unused shared config file(s) found: [/root/.aws/credentials]. If you would like to use them, please update your common-config.toml.
...
2024-01-10T07:33:01Z I! [outputs.cloudwatchlogs] Configured middleware on AWS client
2024-01-10T07:33:01Z I! [logagent] piping log from ubuntu_ec2_auth/auth_count(/home/watch/logs/authCount.log) to cloudwatchlogs with retention -1
2024-01-10T07:33:49Z E! cloudwatch: code: InvalidClientTokenId, message: The security token included in the request is invalid, original error: <nil>
2024-01-10T07:33:49Z W! cloudwatch: 0 retries, going to sleep 180 ms before retrying.
2024-01-10T07:33:49Z E! cloudwatch: WriteToCloudWatch failure, err:  InvalidClientTokenId: The security token included in the request is invalid
	status code: 403, request id: 25441c98-3fd3-42e6-9354-2c795128903d

The error you're seeing in the translator

2024/01/09 08:05:15 E! Credentials path is not set while runasuser is not root

has to do with the shared_credential_file not being set in the common-config.toml when running as a different user.

amazon-cloudwatch-agent/translator/cmdutil/userutil.go

Lines 118 to 122 in ae5042f

    
           if runAsUser != "root" { 
        
           	if _, ok := credentials["shared_credential_file"]; !ok { 
        
           		log.Panic("E! Credentials path is not set while runasuser is not root") 
        
           	} 
        
           }

See https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Agent-commandline-fleet.html#CloudWatch-Agent-profile-instance-first for details on the field.

Aside from the translation issue, I don't believe you will be able to publish across AWS partitions.

	if runAsUser != "root" {
	if _, ok := credentials["shared_credential_file"]; !ok {
	log.Panic("E! Credentials path is not set while runasuser is not root")
	}
	}