Storage: How to fetch the contents of an object?

[marklar]

Thanks for creating gogol! 😄

My goal is to download an object from Storage and write its contents to a file. But I'm stumped by the use of ResumableSources.

Calling download works just fine. It finds the object, yielding a Stream (aka: ResumableSource (ResourceT IO) ByteString).

But I don't know how to work with this Stream. I've tried to use Data.Conduit.($$+-) to put the contents in a file, to no avail. Nothing at all happens.

Here's my code:

fetchFile key = do
  lgr ← newLogger Debug stdout
  env ← newEnv <&> (envLogger .~ lgr) . (envScopes .~ storageReadWriteScope)
  runResourceT . runGoogle env $ do
    -- Fetch file into Stream
    stream ← download (objectsGet bucketName key)
    -- Attempt to write stream into file. Nothing happens.
    return $ stream $$+- sinkFile "./my_file_name"

Could you guide me in the right direction, please?

Thanks!

[brendanhay]

Hi @marklar,

On the last line you've written return $ stream $$+- sinkFile "./my_file_name".

Unfortunately while this probably makes things compile you're actually returning IO (ResourceT IO ()), which indicates the ResourceT action of sinkFile is suspended and hasn't yet been run.

Since the Google monad is an instance of MonadResource from the resourcet package, you can lift the $$+- operator and others from conduit by using liftResourceT.

I've updated the storage example to hopefully make this clearer, but I'll paste it here in its entirety:

fetchFile bucket input output = do
    -- Setup a logger to emit 'Debug' (or higher) log statements to 'stdout':
    lgr  <- Google.newLogger Debug stdout

    -- Create a new environment which will discover the appropriate
    -- AuthN/AuthZ credentials, and explicitly state the OAuth scopes
    -- we will be using below, which will be enforced by the compiler:
    env  <- Google.newEnv <&>
          (Google.envLogger .~ lgr)
        . (Google.envScopes .~ Storage.storageReadWriteScope)

    -- Create a streaming 'Conduit' source of the input file contents:
    body <- Google.sourceBody input

    let key = Text.pack input

    runResourceT . Google.runGoogle env $ do
        -- Upload the 'input' file contents to the specified bucket, using
        -- the file path as the key:
        void $ Google.upload
            (Storage.objectsInsert bucket object' & Storage.oiName ?~ key)
            body

        -- Download from the bucket/key and create a source of the HTTP stream:
        stream <- Google.download (Storage.objectsGet bucket key)

        -- 'Conduit' actions such as '$$+-' need to be lifted into the 'Google'
        -- monad using 'liftResourceT', this will stream the received bytes
        -- into the 'output' file:
        liftResourceT (stream $$+- Conduit.sinkFile output)

[marklar]

Thank you very much, @brendanhay ! The combination of clear explanation plus example code is extremely helpful.

(Silly me... Not thinking clearly about the monadic stack I was dealing with, I had been trying liftIO instead of liftResourceT. I have yet to master monad transformers. 😄 )

And thanks for such a rapid response! Cheers.