s3-pusher does not work if `~/.aws/*` files are missing

Question

s3-pusher does not work if `~/.aws/*` files are missing

systemfreund opened this issue 7 years ago · 9 comments

Case 1

GO_SERVER_DASHBOARD_URL, GO_ARTIFACTS_S3_BUCKET, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION set to correct values in my gocd environment
No ~/.aws/config and ~/.aws/credentials file present
Result:

[go] Task: Plugin with ID: indix.s3publishtook: 7.777s
Error: Interaction with plugin with id 'indix.s3publish' implementing 'task' extension failed while requesting for 'execute'. Reason: [Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.]

Relevant stacktrace:

Caused by: com.amazonaws.SdkClientException: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.
        at com.amazonaws.client.builder.AwsClientBuilder.setRegion(AwsClientBuilder.java:371)
        at com.amazonaws.client.builder.AwsClientBuilder.configureMutableProperties(AwsClientBuilder.java:337)
        at com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46)
        at com.indix.gocd.utils.store.S3ArtifactStore.getS3client(S3ArtifactStore.java:209)
        at com.indix.gocd.utils.store.S3ArtifactStore.<init>(S3ArtifactStore.java:45)
        at com.indix.gocd.s3publish.PublishExecutor.getS3ArtifactStore(PublishExecutor.java:89)
        at com.indix.gocd.s3publish.PublishExecutor.execute(PublishExecutor.java:45)
        at com.indix.gocd.s3publish.PublishTask.handleTaskExecution(PublishTask.java:74)
        at com.indix.gocd.s3publish.PublishTask.handle(PublishTask.java:46)
        at com.thoughtworks.go.plugin.infra.DefaultPluginManager$2.execute(DefaultPluginManager.java:166)
        at com.thoughtworks.go.plugin.infra.DefaultPluginManager$2.execute(DefaultPluginManager.java:161)
        at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.executeActionOnTheService(FelixGoPluginOSGiFramework.java:315)

Case 2

Environment variables set like in Case 1
File ~/.aws/config contains:
```
[default]
region = eu-central-1
```

Result:

Error: Interaction with plugin with id 'indix.s3publish' implementing 'task' extension failed while requesting for 'execute'. Reason: [null]

Relevant stacktrace: (unfortunately it's missing the most interesting part because it's cut-off:

java.lang.RuntimeException: Interaction with plugin with id 'indix.s3publish' implementing 'task' extension failed while requesting for 'execute'. Reason: [null]
        at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:41)
        at com.thoughtworks.go.plugin.access.pluggabletask.JsonBasedTaskExecutor.execute(JsonBasedTaskExecutor.java:41)
        at com.thoughtworks.go.domain.builder.pluggableTask.PluggableTaskBuilder.executeTask(PluggableTaskBuilder.java:97)
        at com.thoughtworks.go.domain.builder.pluggableTask.PluggableTaskBuilder$1.execute(PluggableTaskBuilder.java:74)
        at com.thoughtworks.go.domain.builder.pluggableTask.PluggableTaskBuilder$1.execute(PluggableTaskBuilder.java:71)
        at com.thoughtworks.go.plugin.access.pluggabletask.TaskExtension.execute(TaskExtension.java:57)
        at com.thoughtworks.go.domain.builder.pluggableTask.PluggableTaskBuilder.build(PluggableTaskBuilder.java:71)
        at com.thoughtworks.go.remote.work.Builders.build(Builders.java:75)
        at com.thoughtworks.go.remote.work.BuildWork.execute(BuildWork.java:241)
        at com.thoughtworks.go.remote.work.BuildWork.buildJob(BuildWork.java:206)
        at com.thoughtworks.go.remote.work.BuildWork.build(BuildWork.java:148)
        at com.thoughtworks.go.remote.work.BuildWork.doWork(BuildWork.java:93)
        at com.thoughtworks.go.agent.JobRunner.run(JobRunner.java:60)
        at com.thoughtworks.go.agent.AgentHTTPClientController.retrieveWork(AgentHTTPClientController.java:123)
        at com.thoughtworks.go.agent.AgentHTTPClientController.work(AgentHTTPClientController.java:99)
        at com.thoughtworks.go.agent.AgentController.loop(AgentController.java:78)
        at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273)
        at org.springframework.scheduling.support.MethodInvokingRunnable.run(MethodInvokingRunnable.java:65)
        at org.springframework.scheduling.timer.DelegatingTimerTask.run(DelegatingTimerTask.java:70)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)
Caused by: java.lang.RuntimeException
        at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.executeActionOnTheService(FelixGoPluginOSGiFramework.java:317)
        at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.doOn(FelixGoPluginOSGiFramework.java:245)
        at com.thoughtworks.go.plugin.infra.DefaultPluginManager.submitTo(DefaultPluginManager.java:161)
        at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:32)
        ... 23 more
Caused by: java.lang.StackOverflowError
        at com.google.gson.stream.JsonWriter.writeDeferredName(JsonWriter.java:402)
        at com.google.gson.stream.JsonWriter.value(JsonWriter.java:417)
        at com.google.gson.internal.bind.TypeAdapters$13.write(TypeAdapters.java:362)
        at com.google.gson.internal.bind.TypeAdapters$13.write(TypeAdapters.java:346)
        at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
        at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.write(MapTypeAdapterFactory.java:208)
        at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.write(MapTypeAdapterFactory.java:145)
        at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
        at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.write(ReflectiveTypeAdapterFactory.java:89)
        at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.write(ReflectiveTypeAdapterFactory.java:195)
        at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
        at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.write(ReflectiveTypeAdapterFactory.java:89)
        at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.write(ReflectiveTypeAdapterFactory.java:195)
        at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
        at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.write(ReflectiveTypeAdapterFactory.java:89)
        at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.write(ReflectiveTypeAdapterFactory.java:195)
        at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
        at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.write(ReflectiveTypeAdapterFactory.java:89)
        at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.write(ReflectiveTypeAdapterFactory.java:195)
        at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
        at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.write(ReflectiveTypeAdapterFactory.java:89)
...

Case 3

Environment variables and ~/.aws/config like in Case 2

~/.aws/credentials contains:

[default]
aws_access_key_id = XXX
aws_secret_access_key = ZZZ

Result: now the upload completes successfully

Case 4

Same setting as in Case 3, however change environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY to invalid values

Result: upload still completes successfully

So, to summarize:

If i just set the environment variables it doesn't work
If config and credentials in ~/.aws is missing it doesn't work
If ~/.aws files are present and correct the environment variables have no effect at all

varshavaradarajan commented 7 years ago

v4.0.0

Answer 1 · 2017-06-29T06:06:04.000Z

@systemfreund - Thanks for the detailed issue. Will take a look. Can you please confirm the exact version you are using?

Answer 2 · 2017-06-29T08:24:52.000Z

I am using version 3.1.0 of the plugin and gocd version 17.7.0

Answer 3 · 2017-07-10T19:06:29.000Z

Sorry for the delay - Are you setting the environment variables from Go - ie pipeline level environment variables? If so I have found the bug around that.

Answer 4 · 2017-07-13T06:06:40.000Z

@systemfreund can you please try the v3.2.0 release and let us know how that works out for you?

Answer 5 · 2017-08-31T09:46:34.000Z

Same issue for the material plugin.

Seems to be the case for checking connection for the package material in a pipeline. Even though environment variables are present, the server needs to have the .aws/credentials and .aws/config files.

Answer 6 · 2017-08-31T10:02:35.000Z

@varshavaradarajan - which version of the plugin are you on?

Answer 7 · 2017-09-01T07:11:49.000Z

@varshavaradarajan - thanks for taking the time to go on a call. As confirmed on the call, the s3material plugin is working as intended. Will add a feature to pass credentials and region as package configuration though.

Answer 8 · 2017-11-17T21:15:31.000Z

I am having a similar issue to the above. I have valid creds in the .aws/credentials, and I have .aws/config setup. I also have the same settings in environment variables, however I get the error:

Could not connect to package repository. Reason(s): Interaction with plugin with id 's3material' implementing 'package-repository' extension failed while requesting for 'check-repository-connection'. Reason: [Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.]