Add Ignore counter to job_instance_status and tJobInstanceStop
mattywausb opened this issue · 3 comments
I would like to measure the ignored rows. "Ingnored" means, the incoming information shall not be copied to the destination on purpose. Having this counter one can measure the completnes of the transformation by checking
input = output + reject + ignore
Perhaps we can do that, my problem here is mostly the broken compatibility to existing projects (keep in mind it runs in about 100 projects and not only in ours).
We should break the compatibility only once but than with all changes which we expect.
The table should also not become wide, it will be expensive to read otherwise.
This extention is backwards compatible in the way, that old jobs will still run on the new datamodel. So all you need to do is: deploy new version of datamodel and components.
Today I again stumbled in to the requirement to add ignore counts. Just to keep it in mind, I added this note.