How to implement check rules dynamically in Deequ

[rakesmn]

Hi,
Suppose I have 3 files and I have different check rules for 3 different files. Is there any way I can write the rules in a json/yaml file and implement that in Deequ VerificationSuite dynamically based on the file name instead of creating 3 different check rules for 3 different files ?

Any help would be appreciated.

[saharahsani]

I have this question too..
Is there any way to add a file dynamically in Deequ check rules?

Thanks in advance.

[subham611]

Similar Requirement

Thanks in advance

[dgcaron]

i have a very simple example in pydeequ that only contains a dynamic completeness check but it might help you getting started:

def verify_entity(entity: str): 
    rows = read_entity(entity)
    rules = configuation.collect()
    completeness = Check(spark, CheckLevel.Warning, "Completeness Check")

    suite = VerificationSuite(spark).onData(rows)

    for r in rules:
        if(r.Entity == entity and r.IsRequired):
            completeness = completeness.isComplete(r.Attribute)

    suite.addCheck(completeness) 
    report = suite.run()

    checkResult_df = VerificationResult.checkResultsAsDataFrame(spark, report)

configuation is a dataframe that contains certain rules about an entity that i want to check. With these rules, i use the builder pattern to constuct the checks dynamically.