Support StripPrefix on upload for TransferManager
Opened this issue · 1 comments
Currently transfer manager supports stripping a customer supplied prefix on an object on object download, the same functionality is not supported on upload and the name of the object maintains the full File name as the object name.
Adding scenarios to show some context - need some traction on this
Environment details
OS type and version: MacOS 14.6.1 / Windows 11 (issue happens on both)
Java version: Temurin-11.0.19+7 (build 11.0.19+7)
Version(s): 2.42.0
Steps to reproduce
- Implement the shared code in the documentation to upload multiple files (https://cloud.google.com/storage/docs/uploading-objects#storage-upload-object-java)
- Run the code trying to upload one directory (e.g. /Users/me/source/dir) to some location in a bucket: e.g. gs://my-bucket-name/destination (directory contents are file1.txt, file2.txt) (run the code with arguments
Code example
package org.example;
import com.google.cloud.storage.transfermanager.ParallelUploadConfig;
import com.google.cloud.storage.transfermanager.TransferManager;
import com.google.cloud.storage.transfermanager.TransferManagerConfig;
import com.google.cloud.storage.transfermanager.UploadResult;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List;
import java.util.stream.Stream;
public class Main {
public static void main(String[] args) throws IOException {
uploadDirectoryContents(args[0], Paths.get(args[1]), args[2]);
}
private static void uploadDirectoryContents(String bucketName, Path sourceDirectory, String remotePath) throws IOException {
TransferManager transferManager = TransferManagerConfig.newBuilder().build().getService();
ParallelUploadConfig parallelUploadConfig = ParallelUploadConfig.newBuilder()
.setBucketName(bucketName)
.setPrefix(remotePath)
.build();
List<Path> filePaths = new ArrayList<>();
try (Stream<Path> pathStream = Files.walk(sourceDirectory)) {
pathStream.filter(Files::isRegularFile).forEach(filePaths::add);
}
List<UploadResult> results = transferManager.uploadFiles(filePaths, parallelUploadConfig).getUploadResults();
for (UploadResult result : results) {
System.out.println(
"Upload for "
+ result.getInput().getName()
+ " completed with status "
+ result.getStatus());
}
}
}
Run the above example using arguments
my-bucket-name /Users/me/source/dir destination
I would expect that the bucket contains the following files now
gs://my-bucket-name/destination/file1.txt
gs://my-bucket-name/destination/file2.txt
However, the bucket contains the following (for a MacOS scenario)
gs://my-bucket-name/destination/Users/me/source/dir/file1.txt
gs://my-bucket-name/destination/Users/me/source/dir/file2.txt
For Windows, the /Users/me/source/dir/file1.txt
gets changed to C:\Users\me\source\dir\file1.txt