/xargs_data_demo

demonstration of using xargs and id iteration to process a big ol pile of data

Primary LanguagePythonMIT LicenseMIT

xargs_data_demo

demonstration of using xargs and id iteration to process a big ol pile of data

run

run the whole shebang, splitting up the table into 10 row chunks:

python funcs.py id_ranges 10 | xargs -n 2 python funcs.py get_names

run just one of the functions:

python funcs.py id_ranges 10
python funcs.py get_names 171 181 true

setup

pip install -r requirements.txt

database

dataset taken from https://medium.com/@jiayu./fantasy-premier-league-sqlite-dataset-5b0bfdd052d4. incomplete schema of players table:

CREATE TABLE players (
  id integer,
  photo text,
  web_name text,
  team_code integer,
  status text,
  code integer,
  first_name text,
  second_name text,
  squad_number integer,
  news text,
  now_cost integer,
  news_added text,
  chance_of_playing_this_round integer,
  chance_of_playing_next_round integer,
  value_form real,
  ...

note on stdout and stderr

stdout is used for real output. stderr is used for messaging. python logger writes to stderr. print writes to stdout. Don't use print.