/feed-redis

Container responsible for loading csv data from remote link into redis using awk and redis-mass insert

Primary LanguageAwk

feed-redis

Feeding redis with csv data using awk and redis-mass insert

Docker Hub

Build

  $ docker build -t feed-redis .

Run

  $ docker run --name redis -d redis:3 redis-server --appendonly yes
  $ docker run -d --link redis waleedsamy/feed-redis {DOWNLOADABLE_LINK}

How does it work?

  • downlaod and unzip gz file
  • use awk to build redis SET expression with first columns of the csv as the KEY
  • use mass-insert to insert in redis as cat data.txt | redis-cli --raw --pipe
  • you can use the awk program as awk -F ";" -f tab2json.awk big.csv

Notes:

  • escape single quote in string
   $ echo "Pierre & Vacances Hôtel de l'Esterel" | awk  -v ESQUOTE="\\\'" '{gsub( "[:'\'']",ESQUOTE); print $0;}'
   > Pierre & Vacances Hôtel de l\'Esterel
  • you can find awk tab2json gist here