When pct_invalid=0, no rows should be dropped

Question

When pct_invalid=0, no rows should be dropped

paxcema opened this issue 2 years ago · 3 comments

Answer 1 · 2022-10-21T12:31:13.000Z

Is this bug caused by these two dropnas (https://github.com/mindsdb/type_infer/blob/staging/type_infer/infer.py#L383-L390)?
Because I could not find any other traces of data being skipped related to pct_invalid

Answer 2 · 2022-10-24T13:23:01.000Z

Most likely, yes. I'm not 100% familiar with this feature though (was implemented by another engineer). I do think a test should be part of the fix so that we know it's working as intended in the future.

Answer 3 · 2023-07-04T01:48:52.000Z

Closing, no longer accurate to say pct_invalid has a say in what rows are dropped prior to the type inference step.