A custom implementation of HSET for storing data Memorystore in a cloud dataflow project.
This is a cloud dataflow pipeline code that processes data from a cloud storage bucket, transform it and stores in Google's highly scalable, reduced latency in-memory database called Memorystore which is an implementation of Redis (5.0 version)
Well, it's a pipeline provided by Google to process data to a form we need and it just takes seconds to transform bulk of data. Companies like Spotify and Dailymotion uses this service.
mvn compile exec:java \
-Dexec.mainClass=com.viveknaskar.StarterPipeline \
-Dexec.args="--project=your-project-id \
--jobName=dataflow-custom-redis-job \
--redisHost=127.0.0.1 \
--inputFile=gs://cloud-dataflow-bucket-input/*.txt \
--stagingLocation=gs://cloud-dataflow-pipeline-bucket/staging/ \
--dataflowJobFile=gs://cloud-dataflow-pipeline-bucket/templates/dataflow-custom-redis-template \
--gcpTempLocation=gs://cloud-dataflow-pipeline-bucket/tmp/ \
--runner=DataflowRunner"
Before running the dataflow command, make sure that the storage buckets are created. Storage buckets are global, so no two buckets can be created with the same name.
For checking whether the processed data is stored in the Redis instance after the dataflow pipeline is executed successfully, you must first connect to the Redis instance from any Compute Engine VM instance located within the same project, region and network as the Redis instance.
-
Create a VM instance and SSH to it
-
Install telnet from apt-get in the VM instance
sudo apt-get install telnet
- From the VM instance, connect to the ip-address of the redis instance
telnet instance-ip-address 6379
- Once you are in the redis, check the keys inserted
keys *
https://thedeveloperstory.com/2020/07/28/the-technology-behind-spotify-wrapped-2019/