- Ensure delta files follow as per folder naming convention (DD_MM_YYYY) for today date and yesterday date. Make sure to store csv in each folder . When automating file save, ensure to create a new folder with above naming convention and save each files in it.
Check in
main.pyif csv got any changes on first column nanme as it would affect changes tracking in line 14 where key = "col1" Change col1 to new first column name
- Run the program when both file are in respective folder . Manipulate the path of your files for today and yesterday for comparison in val.properties files.
- Running the program process: -
In terminal,
- create new python environment
- Activate the environment
- Run
pip install requirements.txtto install all packages dependencies - Run the program
python main.py - Check the log files for any error if any
- Check the delta folder for specific changes tracking
- To automate the process, ive create a script to automate the programm using shell script.
In terminal
chmod +x run_script.shcrontab -e0 0 * * * /path/to/run_script.sh >> /path/to/script_log.log 2>&1- Run on midnight and if failed schedule to run again another 4 hours until succeed.