showing Incorrect Missing data in HTML Report

Question

showing Incorrect Missing data in HTML Report

Opened this issue 5 years ago · 6 comments

harika1419 commented 5 years ago

After generating the HTML report using spark-df- profiling It is showing the percentage of Missing data as 0%.

Even though dataframe has some missing data

Answer 1 · 2019-07-31T15:06:32.000Z

Could you give an example?

Answer 2 · 2020-01-07T14:54:49.000Z

Is this fixed yet? mine also shows wrong missing data as 0%

Answer 3 · 2020-01-08T06:43:07.000Z

@harika1419 I think I found the issue. It's in line 397.
Change to this:

results_data = df.select(column).na.drop().agg(countDistinct(col(column)).alias("distinct_count"),
                                                       count(col(column)).alias("count")).toPandas()

@julioasotodv you might need to look at this solution

Answer 4 · 2020-01-08T11:32:37.000Z

Hi...
That issue was fixed after upgrading the spark from 1.6 to 2.3.3

Answer 5 · 2020-01-08T13:00:55.000Z

Hi @harika1419,
Thanks for informing. I'm facing this issue while using spark 2.4.2, that is why I thought its not fixed yet.

Answer 6 · 2021-06-07T14:01:28.000Z

I'm on Spark 3.1.0, and it's showing wrong. Also the number of zeros are wrong.