VStore is a data store for supporting fast, resource efficient analytics over large archival videos. Please read our paper in EuroSys '19: VStore: A Data Store for Analytics on Large Videos and visit out website for more details.
This source code orchestrates VStore's ingestion, storage, retrieval, and consumption based on 2 modern computer vision pipelines, OpenALPR and NoScope. VStore's configurations, including the derivation of consumption formats, the derivation of storage formats, and data erosion are not included.
In this repo, for the codebases (two sinks) inherited from third-parties, i.e., OpenALPR and NoScope, are NOT provided.
To build VStore, you need the following to be installed
- ZeroMQ 2.2.x
- lmdb
Follow the requirement of OpenALPR and NoScope.
Before building VStore, merge the sink towards the current code:
For OpenALPR, under ./VStore-OpenALPR, donwload OpenALPR; merge ./VStore-OpenALPR/src into OpenALPR src directory. For NoScope, under ./VStore-NoScope/vstore, donwload OpenALPR; merge ./VStore-NoScope/vstore/src into OpenALPR src directory; Under ./VStore-NoScope/, download NoScope code from their repo, and follow their instructions for details.
cd ./src
source env-teddy.sh
mkdir cmake-build-debug; cd cmake-build-debug; cmake .. -DCMAKE_BUILD_TYPE=Debug; cd ..
mkdir cmake-build-release; cd cmake-build-release; cmake .. -DCMAKE_BUILD_TYPE=Release; cd ..
build-all
The first step is to transcode the videos to the target formats: Follow the guides here.
To generate raw video footage,
ffmpeg -i input.mp4 -c:v rawvideo -pix_fmt yuv420p video-raw.yuv
To generate encoded video chunks,
ffmpeg -i input.mp4 -acodec copy -f segment -vcodec copy -reset_timestamps 1 -map 0 ./video-chunks/output%04d.mp4
Build the footage to lmdb:
for raw video footage,
/tmp/teddyxu/Debug/test-db-build.bin -r --dpath=/path/to/database --vpath=/path/to/video-raw.yuv width height
for encoded video chunks,
/tmp/teddyxu/Debug/test-db-build.bin -e --dpath=/path/to/database --vpath=/path/to/video-chunks width height
/tmp/teddyxu/Debug/test-source.bin
/tmp/teddyxu/Debug/decode-serv.bin
For OpenALPR, revoke OpenALPR sink by running
/tmp/teddyxu/Debug/test-sink.bin
For NoScope, follow their instructions and configure the data directory. After building NoScope sink, revoke the sink by running sample a command like
./VStore-NoScope/data/experiments/jackson-town-square/0.02/run_testset.sh 0 VIDEO_LEN RESOLUTION