weavejester/clj-aws-s3

S3-client gets GC'd

arohner opened this issue · 4 comments

arohner commented

When downloading large objects with get-object, I reliably get "socket is closed". This is caused because the (s3-client cred) is GC'd before my file is done downloading.

See https://forums.aws.amazon.com/thread.jspa?threadID=83326, the reply by Yosuke on Jan 23,2012 3:46 PM for coroboration.

I'm doing

(let [resp (s3/get-object bucket key)]
  (io/copy (-> resp :content) (io/file dest-path))))

My download takes several minutes to finish, and I reliably get the "socket is closed" less than a minute into the download. Replacing (s3/get-object) with its source code, and keeping a reference on the s3-client object fixes the bug.

weavejester commented

Thanks for coming up with a cause for this. It couldn't have been easy!

Memoizing the s3-client function should, in theory, fix this problem...

weavejester commented

I've pushed a version to clojars with a memoized s3-client.

Try changing the version of your clj-aws-s3 dependency to "0.2.2-SNAPSHOT" and let me know if that solves your problem.

arohner commented

Finally got a chance to test this! Yes, 2.2-SNAPSHOT fixes the issue for me.

weavejester commented

Okay, I've released 0.2.2. That should fix this issue.