googleapis/java-storage

Support StripPrefix on upload for TransferManager

Opened this issue · 1 comments

Currently transfer manager supports stripping a customer supplied prefix on an object on object download, the same functionality is not supported on upload and the name of the object maintains the full File name as the object name.

Adding scenarios to show some context - need some traction on this

Environment details

OS type and version: MacOS 14.6.1 / Windows 11 (issue happens on both)
Java version: Temurin-11.0.19+7 (build 11.0.19+7)
Version(s): 2.42.0

Steps to reproduce

  1. Implement the shared code in the documentation to upload multiple files (https://cloud.google.com/storage/docs/uploading-objects#storage-upload-object-java)
  2. Run the code trying to upload one directory (e.g. /Users/me/source/dir) to some location in a bucket: e.g. gs://my-bucket-name/destination (directory contents are file1.txt, file2.txt) (run the code with arguments

Code example

package org.example;

import com.google.cloud.storage.transfermanager.ParallelUploadConfig;
import com.google.cloud.storage.transfermanager.TransferManager;
import com.google.cloud.storage.transfermanager.TransferManagerConfig;
import com.google.cloud.storage.transfermanager.UploadResult;

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List;
import java.util.stream.Stream;

public class Main {
    public static void main(String[] args) throws IOException {
        uploadDirectoryContents(args[0], Paths.get(args[1]), args[2]);
    }

    private static void uploadDirectoryContents(String bucketName, Path sourceDirectory, String remotePath) throws IOException {
        TransferManager transferManager = TransferManagerConfig.newBuilder().build().getService();
        ParallelUploadConfig parallelUploadConfig = ParallelUploadConfig.newBuilder()
                .setBucketName(bucketName)
                .setPrefix(remotePath)
                .build();

        List<Path> filePaths = new ArrayList<>();
        try (Stream<Path> pathStream = Files.walk(sourceDirectory)) {
            pathStream.filter(Files::isRegularFile).forEach(filePaths::add);
        }
        List<UploadResult> results = transferManager.uploadFiles(filePaths, parallelUploadConfig).getUploadResults();

        for (UploadResult result : results) {
            System.out.println(
                    "Upload for "
                            + result.getInput().getName()
                            + " completed with status "
                            + result.getStatus());
        }
    }
}

Run the above example using arguments
my-bucket-name /Users/me/source/dir destination

I would expect that the bucket contains the following files now

gs://my-bucket-name/destination/file1.txt
gs://my-bucket-name/destination/file2.txt

However, the bucket contains the following (for a MacOS scenario)

gs://my-bucket-name/destination/Users/me/source/dir/file1.txt
gs://my-bucket-name/destination/Users/me/source/dir/file2.txt

For Windows, the /Users/me/source/dir/file1.txt gets changed to C:\Users\me\source\dir\file1.txt