synthetichealth/synthea

Vital signs must have finite values - NaN is invalid

arvindshmicrosoft opened this issue · 5 comments

arvindshmicrosoft commented

What happened?

I got this exception java.lang.IllegalArgumentException: Vital signs must have finite values - NaN is invalid multiple times with a build from commit 6192ed7 (latest as of 2023-10-07). Symptomatically it's identical to #1038.

A sample command line was:

java -Xmx24G -jar ./build/libs/synthea-with-dependencies.jar --exporter.fhir.export false --exporter.csv.export true --exporter.baseDirectory '/synthea_data/Arizona Phoenix 706' --generate.log_patients.detail none --exporter.clinical_note.export false -p 781512 Arizona Phoenix

Relevant output from the process:

Running with options:
Population: 781512
Seed: 1696672364458
Provider Seed:1696672364458
Reference Time: 1696672364458
Location: Phoenix, Arizona
Min Age: 0
Max Age: 140

As I mentioned, the exception is observed in many runs, for different cities / areas. For example, I ran all the cities in California, the setVitalSign / NaN exception is seen in the following cities. How impactful is this exception? Is the entire generated dataset suspect, or will the impact be localized only to specific individual?

California Anaheim (2 occurrences)
California Arden-Arcade
California Bakersfield
California Blythe
California Concord
California Davis
California El Monte
California Fresno
California Inglewood
California Irvine
California Menlo Park
California Moreno Valley
California Orinda
California Oxnard
California San Diego (2 occurrences)
California San Jose
California Santa Ana
California Santa Monica
California Soledad
California Stanton
California Vallejo
California Westmont
California Windsor

Environment

OS:

Linux version 4.18.0-425.10.1.el8_7.x86_64 (mockbuild@dal1-prod-builder001.bld.equ.rockylinux.org) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-15) (GCC)) #1 SMP Thu Jan 12 16:32:13 UTC 2023

Java:

openjdk 11.0.20 2023-07-18 LTS
OpenJDK Runtime Environment (Red_Hat-11.0.20.0.8-1) (build 11.0.20+8-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-11.0.20.0.8-1) (build 11.0.20+8-LTS, mixed mode, sharing)

Relevant log output

java.lang.IllegalArgumentException: Vital signs must have finite values - NaN is invalid
        at org.mitre.synthea.world.agents.Person.setVitalSign(Person.java:486)
        at org.mitre.synthea.modules.WeightLossModule.manageYearTwoThroughFive(WeightLossModule.java:165)
        at org.mitre.synthea.modules.WeightLossModule.process(WeightLossModule.java:82)
        at org.mitre.synthea.engine.Generator.updatePerson(Generator.java:712)
        at org.mitre.synthea.engine.Generator.createPerson(Generator.java:668)
        at org.mitre.synthea.engine.Generator.generatePerson(Generator.java:478)
        at org.mitre.synthea.engine.Generator.lambda$run$3(Generator.java:383)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
eedrummer commented

Hey @arvindshmicrosoft, thanks for reporting this. I'm looking into it.

As far as the exception goes, that crashes the executor for the single patient. It should not impact the rest of the population.

  • 👍1
eedrummer commented

Hi @arvindshmicrosoft! I'm having a hard time reproducing this issue. Two questions:

  1. Are you only seeing the NaN issue? I have seen a few other stack traces while running simulations, but not this one.
  2. Could you run unzip -p synthea-with-dependencies.jar META-INF/MANIFEST.MF and put the output in a comment? I want to make sure we are working from the exact same jar file.
arvindshmicrosoft commented

Hi @arvindshmicrosoft! I'm having a hard time reproducing this issue. Two questions:

  1. Are you only seeing the NaN issue? I have seen a few other stack traces while running simulations, but not this one.
  2. Could you run unzip -p synthea-with-dependencies.jar META-INF/MANIFEST.MF and put the output in a comment? I want to make sure we are working from the exact same jar file.

Sure, here we go @eedrummer. Note that I have built from source - commit 6192ed7 (latest as of 2023-10-07).

Manifest-Version: 1.0
Main-Class: App
Build-Timestamp: 2023-10-06T21:17:14.165+0000
Build-Version:
Created-By: Gradle 8.2.1
Build-Jdk: 11.0.20 (Red Hat, Inc. 11.0.20+8-LTS)
Build-OS: Linux amd64 4.18.0-477.27.1.el8_8.x86_64

Unfortunately, I don't seem to have the old logs available. I found some other runs - where the same exception is reported. There are multiple cities for which this was observed, but as a representative example, I am sharing the logs for Austin, TX. You'll notice some other exceptions in this as well, but there are other cities with just the setVitalSign exception, like this one for Wauwatosa, WI.

eedrummer commented

@arvindshmicrosoft thanks for the update. The log for Wauwatosa, Wisconsin was especially helpful because it allowed me to reproduce the issue within a reasonable sized population.

The root of the issue is a method that converts BMI percentile into an actual BMI for an individual. For the case causing the issue in Wauwatosa, the individual has a BMI in the ~99.74th percentile. Given other parameters in the equation, this high value caused a component to flip from positive to negative, which causes the entire method to return NaN. The issue is somewhat rare because the code path also requires that the simulated individual undergoes successful weight loss in their late teenage years and then regress in weight management where their period of weight regression overlaps their 20th birthday. This is where the code switches from pediatric to adult weight management algorithms.

I should be able to put together a PR to fix this particular issue next week.

  • 🎉1
arvindshmicrosoft commented

@eedrummer Thank you for fixing this issue. I no longer see this issue with the current code.