Combining with set_output can produce errors
JulienRoussel77 opened this issue · 1 comments
Expected Behavior
Calling set_output(transformer="pandas") should have no impact on a OneHotEncoder, since the outputs are already dataframes.
Actual Behavior
The OneHotEncoder develops an inconsistent behavior, producing errors in the case of subsequent fit_transform calls.
Steps to Reproduce the Problem
df = pd.DataFrame({"C1": ["a", "a"], "C2": ["c", "d"]})
ohe = OneHotEncoder().set_output(transform="pandas")
ohe.fit_transform(df.iloc[:1]) # commenting this line avoids the error
ohe.fit_transform(df.iloc[:2]) # this line produces an error
ValueError: Length mismatch: Expected axis has 3 elements, new values have 2 elements
Specifications
pandas : 2.0.1
sklearn : 1.3.2
category_encoders : 2.6.3
I think I see what's happening. In BaseEncoder.fit
,
category_encoders/category_encoders/utils.py
Lines 323 to 324 in 06e46db
we set
feature_names_out_
by looking at the encoder's transformed output (L324). But when sklearn is set to produce pandas output, it uses that attribute if available at L323. So it uses the old attribute. Depending on how sklearn picks its column names, we may be able to just delete the attribute somewhere earlier in fit
. This probably affects all the encoders, too, not just OHE.
Update: I tried it out, and the error doesn't get thrown for most other encoders, because they don't change the number of columns; BinaryEncoder
similarly fails. But even for other encoders, making the second fit_transform
on a dataframe with new column names ends up using the old column names. Adding just self.feature_names_out_ = None
before L323 above fixes the problem in all cases, PR to come.