Cannot POST to S3 backup bucket in us-west-2

[pdehlke]

priam 1.2.29, running successfully in us-east-1. When I copy my AMI (which is fully baked with all the tomcat/priam/cassandra bits, no post-boot customizations are done) to us-west-2, priam is no longer able to create cassandra backups in S3:

014-05-28 02:32:39.0204 DEBUG DefaultQuartzScheduler_Worker-1 com.amazonaws.request Received error response: Status Code: 301, AWS Service: null, AWS Request ID: 596FB7D4F3E8BA0B, AWS Error Code: PermanentRedirect, AWS Error Message: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint., S3 Extended Request ID: wSJg0H74J/dKeOBTqJBdm6o/AINZJWLHXBoT890pCvFXKaEritb0ms72ITOtLeLrTi37BEtA7XE=

From the logs, it appears to me that priam is inserting a space in the URI it attempts to POST to:

014-05-28 02:32:39.0190 DEBUG DefaultQuartzScheduler_Worker-1 com.amazonaws.request Sending Request: POST https://REDACTED-cassandra-backups
.s3-us-west-2.amazonaws.com /cassandra_backups%2Fus-west-2%2Fcassandra%2F14178431955039102644307275311624381702%2F201405280204%2FSST%2Fsyst
em%2Fschema_keyspaces%2Fsystem-schema_keyspaces-jb-1-Data.db Parameters: (uploads: null, ) Headers: (Authorization: AWS REDACTED:REDACTED, Date: Wed, 28 May 2014 02:32:39 GMT, User-Agent: aws-sdk-java/1.3.27 Linux/3.13.0-24-generic Java_HotSpot(T
M)_64-Bit_Server_VM/24.51-b03, Content-Type: application/x-www-form-urlencoded; charset=utf-8, )

Am I misreading the log?

[pdehlke]

By the way, this error seems to leak file handles; eventually priam starts to fail with:

014-05-28 03:07:58.0447 WARN DefaultQuartzScheduler_Worker-8 com.netflix.priam.utils.CassandraMonitor Exception thrown while checking if C
assandra is running or not
java.io.IOException: Cannot run program "pgrep": error=24, Too many open files
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1041)
at java.lang.Runtime.exec(Runtime.java:617)
at java.lang.Runtime.exec(Runtime.java:450)
at java.lang.Runtime.exec(Runtime.java:347)
at com.netflix.priam.utils.CassandraMonitor.execute(CassandraMonitor.java:53)
at com.netflix.priam.scheduler.Task.execute(Task.java:93)
at org.quartz.core.JobRunShell.run(JobRunShell.java:199)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:546)
Caused by: java.io.IOException: error=24, Too many open files
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.(UNIXProcess.java:135)
at java.lang.ProcessImpl.start(ProcessImpl.java:130)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1022)
... 7 more

[timiblossom]

Hi pdehlke ,

I think this could be due to S3 permission setting. Yes, because Priam can't connect to S3, it leaks out the file descriptors. Do you want to submit a fix for this?

Thanks.

[pdehlke]

Not at all clear that it's S3 permissions. Again, the same AMI is able to back up to S3 when it's run in us-east-1, but not when it's run in us-west-2. I wrote this script that tests the credentials that priam is using:

#!/bin/sh
rm -rf $HOME/.aws
mkdir -p $HOME/.aws
touch us-west-2-test

. /etc/awscredential.properties

cat > $HOME/.aws/config <<EOF
[default]
aws_access_key_id = $AWSACCESSID
aws_secret_access_key = $AWSKEY
EOF

echo "EC2 availability zone is `curl http://169.254.169.254/latest/meta-data/placement/availability-zone -s `"
echo
echo "Listing Priam S3 directory: "
aws s3 ls s3://REDACTED-cassandra-backups/cassandra_backups/
echo "uploading to s3..."
echo
aws s3 cp us-west-2-test s3://REDACTED-cassandra-backups/cassandra_backups/
echo "Listing Priam S3 directory: "
aws s3 ls s3://REDACTED-cassandra-backups/cassandra_backups/

And the results are:

root@ip-10-253-134-124:~# ./setup_aws.sh
EC2 availability zone is us-west-2a

Listing Priam S3 directory:
                           PRE us-east-1/
uploading to s3...

upload: ./us-west-2-test to s3://REDACTED-cassandra-backups/cassandra_backups/us-west-2-test
Listing Priam S3 directory:
                           PRE us-east-1/
2014-05-30 21:05:01          0 us-west-2-test

So there's something else going on here.

Thanks

[tulumvinh]

Hi pdehlke,

Base on the original stack trace you provided, AWS complains about the endpoint. You mentioned that there is a blank in the generated uri, how are you providing the host and bucket name to Priam?

Thanks