GcsLib
GcsLib is the libraries of the Google cloud storage, The main purpose is to support the transfer of large files.
Project Goals
support read and write large file
Features and Updates:
- New extension class of the InputStream and OutputStream. add a new cache layer and is optimized for the blob read and write.
- The contents are pre-cached by chunks when there is read operation. This will eliminate the times of http request and increase the performance greatly.
- resumeable uploads are used for the write operation. Data is buffered firstly and then be uploaded if the buffer size exceed the threshold. This also can eliminate the times of http request and increase the performance greatly.
How To use
1.include the gcslib in your project, initialize the config
Config config = new Config("gcp_project_id", "gcp_service_account_json_file");
2.read from the blob
GcsReqParmas insParams = new GcsReqParmas();
insParams.setBucket("xxxxx.appspot.com");
insParams.setBlob("xxxx-2020-03-25.log");
BlobBufferedIns bbIns = new BlobBufferedIns(insParams);
String line;
while ((line = bbIns.readLine()) != null)
{
System.out.println((line));
Thread.sleep(100);
}
bbIns.close()
3.Write to the blob
GcsReqParmas ousParams = new GcsReqParmas();
ousParams.setBucket("xxxxx.appspot.com");
ousParams.setBlob("xxxx-2020-03-25.log");
BlobBufferedOus bbOus = new BlobBufferedOus(config, outParams);
String outLine = "insert new line here";
bbOus.writeLine(outLine);
bbOus.close();
4.please find other functions from the source code.
Performance Test
- The performance depends on the machine and the network.
Dependency
- [google-cloud-storage] Google Cloud Storage Library for Java .
Limitation and known issues:
- no file verification
Supported platforms
- Linux
- MacOS
- windows
License
Copyright (C) 2017 Wesley Wu jie1975.wu@gmail.com
This code is licensed under The General Public License version 3
FeedBack
Your feedbacks are highly appreciated! :)