Refactor data processing pipeline.
Closed this issue · 0 comments
ProjectsByJackHe commented
As of right now, we generate the .sql file dynamically and have a simple python script batch execute all the SQL files generated from the test automation to save data to the database / update other state.
A new proposal for a better data architecture is to offload the generation of the sql script to the python script, and have intermediary JSON files be used.
Benefits of this approach is we have more control over how we generate this SQL script, and have that separation with the powershell script that actually runs the jobs.
This is a P2 right now as we don't have logic that is super complicated that needs to be generated at the end.