cqframework/cql-execution

Issue with CQL engine

abhishekctv opened this issue · 7 comments

abhishekctv commented

Details

We're currently using the cql-execution library and facing some issues. We're trying to get the NCQA BCS-E certification. We have already created a system that uses this library to predict the numerator and exclusion. Prediction for the numerator is working fine with no issue; however, exclusion prediction is causing some problems.

We've checked the library and we believe we have identified the issue as follows.

In the main CQL file, the following code seems to be causing some issues with the cql-execution.

include FHIRHelpers version '4.0.1' called FHIRHelpers
include NCQA_HealthPlanEnrollment version '1.1.0' called Enrollment
include NCQA_Status version '1.1.0' called Status
include NCQA_FHIRBase version '1.1.0' called FHIRBase
include NCQA_Hospice version '1.1.0' called Hospice
include NCQA_AdvancedIllnessandFrailty version '1.1.0' called AdvancedIllnessFrailty
include NCQA_PalliativeCare version '1.1.0' called PalliativeCare

We believe the include functionality is not working.

Why we think that is the case

The exclusion rule defined in the CQL is provided below. Here in the rules only Mastectomy Exclusion is working as it is directly defined in the main CQL file, whereas the rest of the conditions are not working as those are defined in other files and linked using the "include" functionality.

define "Exclusions":
  Hospice."Hospice Intervention or Encounter"
    or "Mastectomy Exclusion"
    or AdvancedIllnessFrailty."Advanced Illness and Frailty Exclusion Not Including Over Age 80"
    or PalliativeCare."Palliative Care Overlapping Period" ( "Measurement Period" )

Note: The Numerator is causing no issues as it completely defined in the same file.

Details on our setup

We are using version 3.0.1 of cql-execution and 2.1.5 of cql-exec-fhir (We're converting the NCQA Sample Deck to FHIR).
Nodejs: v16.9.1
OS: Windows 11
Valueset: BCSE_HEDIS_MY2022-1.1.0
Code for running the engine:

const measureData = JSON.parse(await fs.readFile(c.MEASURE_TO_READ, "utf-8"))

        const libraryFilename = (await fs.readdir(c.LIBRARIES_TO_READ)).filter(filename => path.extname(filename) === ".json")

        const libraryJson = await Promise.all(libraryFilename.map(async filename => JSON.parse(await fs.readFile(path.join(c.LIBRARIES_TO_READ, filename), "utf-8"))))

        const processedLibraries: { [index: string]: any } = {};
        libraryJson.forEach(item => {
            processedLibraries[item.library.identifier.id] = item;
        })

        const valuesetFilenames = (await fs.readdir(c.VALUESETS_TO_READ)).filter(filename => path.extname(filename) === ".json")
        const valuesetJson: R4.IValueSet[] = await Promise.all(valuesetFilenames.map(async filename => JSON.parse(await fs.readFile(path.join(c.VALUESETS_TO_READ, filename), "utf-8"))))

        const processedValuesets: { [url: string]: { [version: string]: R4.IValueSet_Contains } } = {};
        valuesetJson.forEach(item => {
            if (item.url && item.version && item.expansion?.contains) {
                // @ts-expect-error
                processedValuesets[item.url] = {
                    [item.version]: [...item.expansion?.contains]
                };
            }
        })

        const parameters = {
            'Measurement Period': new Interval(START_INTERVAL, END_INTERVAL, true, false)
        }

        const executor = new Executor(new Library(measureData, new Repository(processedLibraries)), new CodeService(processedValuesets), parameters)

        const patientSource = cqlfhir.PatientSource.FHIRv401();
        patientSource.loadBundles([patientData]);

Can you please have a look at the issue and tell us what is wrong? Are you doing something wrong or there is some issue with the CQL (and elm files) or there is some issue with the cql-execution library.

birick1 commented

Hi @abhishekctv.

I have some experience on this topic and made the demo at https://www.ncqa.org/videos/ncqa-digital-measure-execution-demo/ while I was at NCQA. That demo uses BCSE as an example.

To start, here's a couple confirmations:

  • The code you've provided looks like an async/await version of the code from that demo. I reviewed and did some testing, and your code works as expected. No issue there.
  • I can also confirm both NCQA and MITRE test measures for working correctly with libraries. No issue there.

Since NCQA's measures are proprietary, I won't be able to replicate the issue you are having. Nonetheless, I have some ideas and suggestions.

  1. In the NCQA package, there should be a README.txt that includes information about the version of cql-execution and cql-exec-fhir used to test the measure at NCQA. If there isn't a README.txt, reach out to NCQA for guidance.

    • For example, in the demo linked above, you'll see me point out cql-execution 2.2.0 and cql-exec-fhir 1.5.0.
    • Once you have engine version info for your package, downgrade appropriately and try your code above.
    • If downgrading leads to a correct result, then work up through the versions to find the one at which it doesn't work. In the last two years, there have been numerous authoring pattern changes, cql-to-elm transpilation updates, and cql-execution updates tightening up the spec, resolving issues, etc. This could help identify.

  2. You mentioned you are converting NCQA's sample deck to FHIR. That can be particularly tricky to do. It's possible there is a conversion error that could end up looking like a measure error as above. It might be helpful to reach out to NCQA to see if they have suggestions or resources to help.

abhishekctv commented

Issue with "Encounter Has Diagnosis" Function

Hi @birick1,

I wanted to thank you for the informative video you provided; it has been instrumental in our work. However, we have encountered an issue with our FHIR converter that we have been unable to resolve completely.

The problem arises when we are attempting to work with the condition "Acute Inpatient Encounter with Advanced Illness." Below is the relevant CQL:

define "Acute Inpatient Encounter with Advanced Illness":
  exists ( ( Status."Finished Encounter" ( [Encounter: "Acute Inpatient"] ) ) InpatientEncounter
      where Encounters."Encounter Has Diagnosis" ( InpatientEncounter, [Condition: "Advanced Illness"] )
        and date from start of FHIRBase."Normalize Interval" ( InpatientEncounter.period ) during Interval[date from start of "Measurement Period" - 1 year, date from 
        end of "Measurement Period"]
  )

define function "Encounter Has Diagnosis"(Encounter FHIR.Encounter, Conditions List<FHIR.Condition>):
  AnyTrue((Encounter.diagnosis D
      return D.condition.reference)CRef
      return exists(Conditions C
          where C.id = FHIRBase."GetId"(CRef)
      )
  )

define function "GetId"(uri String):
  if ( PositionOf('/', uri)> 0 ) then Last(Split(uri, '/'))
    else uri

We suspect that the issue lies in the "Encounter Has Diagnosis" function, specifically with the part that connects the Encounter with the Condition (D.condition.reference). To investigate further, we have provided sample FHIR JSON for an Encounter and a Condition:

Sample Encounter JSON:

{
    "resource": {
        "resourceType": "Encounter",
        "id": "1e09dd45-577b-11ee-9f66-38d57ad07149",
        "status": "finished",
        "class": {
            "system": "http://www.ama-assn.org/go/cpt",
            "version": "2022.2.21AB",
            "code": "99233",
            "display": "Subsequent hospital care, per day, for the evaluation and management of a patient, which requires at least 2 of these 3 key components: A detailed interval history; A detailed examination; Medical decision making of high complexity. Counseling and/or coordination of care with other physicians, other qualified health care professionals, or agencies are provided consistent with the nature of the problem(s) and the patient's and/or family's needs. Usually, the patient is unstable or has developed a significant complication or a significant new problem. Typically, 35 minutes are spent at the bedside and on the patient's hospital floor or unit."
        },
        "type": [
            {
                "coding": [
                    {
                        "system": "http://www.ama-assn.org/go/cpt",
                        "version": "2022.2.21AB",
                        "code": "99233",
                        "display": "Subsequent hospital care, per day, for the evaluation and management of a patient, which requires at least 2 of these 3 key components: A detailed interval history; A detailed examination; Medical decision making of high complexity. Counseling and/or coordination of care with other physicians, other qualified health care professionals, or agencies are provided consistent with the nature of the problem(s) and the patient's and/or family's needs. Usually, the patient is unstable or has developed a significant complication or a significant new problem. Typically, 35 minutes are spent at the bedside and on the patient's hospital floor or unit."
                    }
                ]
            }
        ],
        "period": {
            "start": "2023-09-23T03:35:21+00:00Z"
        },
        "subject": {
            "reference": "Patient/1da00559-577b-11ee-b33e-38d57ad07144"
        },
        "diagnosis": [
            {
                "condition": {
                    "reference": "Condition/1e09dd48-577b-11ee-9f6f-38d57ad07144"
                },
                "use": {
                    "coding": [
                        {
                            "system": "http://hl7.org/fhir/sid/icd-10-cm",
                            "code": "C78.4",
                            "display": "ICD-10"
                        }
                    ]
                },
                "rank": 1
            }
        ]
    }
},

Sample Condition JSON:

{
    "resource": {
        "resourceType": "Condition",
        "id": "1e09dd48-577b-11ee-9f6f-38d57ad07144",
        "clinicalStatus": {
            "coding": [
                {
                    "system": "http://terminology.hl7.org/CodeSystem/condition-clinical",
                    "code": "active"
                }
            ]
        },
        "code": {
            "coding": [
                {
                    "system": "http://hl7.org/fhir/sid/icd-10-cm",
                    "version": "2022.1.21AA",
                    "code": "C78.4",
                    "display": "Secondary malignant neoplasm of small intestine"
                }
            ]
        },
        "subject": {
            "reference": "Patient/1da00559-577b-11ee-b33e-38d57ad07144"
        },
        "onsetDateTime": "2023-05-21T00:00:00+00:00",
        "abatementDateTime": "2023-05-21T00:00:00+00:00",
    }
}

Could you please check the FHIR data help us identify where we made the mistake in converting the data? If you require any additional information or clarification, please don't hesitate to ask.

Thank you for your assistance in resolving this issue.

birick1 commented

Hi @abhishekctv

To eliminate variables, here's my setup to test the CQL above:

  • cql-execution 2.4.0.
  • cql-exec-fhir 2.1.5
  • Compiled the CQL to ELM with cql-to-elm 2.5.0.
  • I don't have the code, so I created my own versions of Status."Finished Encounter" and FHIRBase."Normalize Interval".
  • I don't have the valuesets, so I changed [Encounter: "Acute Inpatient"] to just [Encounter] and [Condition: "Advanced Illness"] to just [Condition]. This will work as expected since we're only including the resources above.
  • Measurement period: 1/1/2022 to 1/1/2024
  • I created a basic patient resource to coincide with the Encounter/Condition resources above.

With that, I found two errors in the JSON:

  • Remove the , after "abatementDateTime": "2023-05-21T00:00:00+00:00",. I get an Unexpected token } error during execution if it is included.
  • The format used for datetime is incorrect.

Correcting both of those, the definition "Acute Inpatient Encounter with Advanced Illness" is returning the expected result for these resources (true).

image

It's worth noting, incorrect json datetime formats lead to a null datetime (unknown datetime in CQL). It looks like this in the engine:
image

With the correct json format, encounter.period.start looks like this in the engine:
image

birick1 commented

This is another way to resolve:

  • Incorrect: "start": "2023-09-23T03:35:21+00:00Z"
  • Correct (exclude the Z): "start": "2023-09-23T03:35:21+00:00"

I thought I noticed the other dates including Z at the end, but looking back they don't. So I think onsetDateTime and abatementDateTime are ok. Will update prior comment.

Overall, I'd suggest using a consistent format across all datetimes.

abhishekctv commented

Thank you, @birick1, as you suspected, the problem stemmed from the DateTime format. We've managed to address the majority of the issues, and everything is now functioning as anticipated.

I have a final inquiry: Is there a method to print the results from the internal CQL files, such as the output from NCQA_AdvancedIllnessandFrailty.Has Criteria Indicating Frailty? Presently, the output file generated by cql-execution solely includes results from the BCSE_HEDIS_MY2022-1.1.0 file.

birick1 commented

Hi @abhishekctv. I'm glad that resolved the issue.

For your inquiry, there's two options I can think of depending on what you might want to do.

Assume const result = await executor.exec(patientSource)

1. Modify Measure CQL
Execution results for a patient are in result.patientResults.<patientId>. As you noted, only the definition results in the measure file itself are surfaced here. Since NCQA bundles include the CQL, one option is to modify the BCSE file with a new definition that echoes the library result, and then generate new ELM. It would look something like this:

define "Echoed Criteria": NCQA_AdvancedIllnessandFrailty."Has Criteria Indicating Frailty"

The evaluation of the library definition would then show up in result.patientResults.<patientId>.Echoed Criteria

2. Use ELM Id
In the ELM for the NCQA_AdvancedIllnessandFrailty library, search by name for "Has Criteria Indicating Frailty". There should be a block in the ELM representing the CQL definition. It will have fields like localId, locator, resultTypeName, etc. The important field to make note of is localId. Once you've found and made a note of the localId, look for

result.localIdPatientResultsMap.<patientId>.NCQA_AdvancedIllnessandFrailty.<localId>

That field holds the result of NCQA_AdvancedIllnessandFrailty.Has Criteria Indicating Frailty.

birick1 commented

Closing this ticket since original issue is resolved.