`SingleColumnTransformer`s don't work with `ColumnTransformer`

[Vincent-Maladiere]

Describe the bug

On the dev version, both GapEncoder and MinHashEncoder work well with the TableVectorizer, but fail with the ColumnTransformer from scikit-learn.

Steps/Code to Reproduce

from sklearn.compose import make_column_transformer
from skrub.datasets import fetch_employee_salaries
from skrub import GapEncoder

bunch = fetch_employee_salaries()

transformer = make_column_transformer(
    (GapEncoder(), bunch.X.select_dtypes("object").columns)
)
transformer.fit(bunch.X)

Expected Results

No error is thrown

Actual Results

ValueError: GapEncoder.fit should be passed a single column, not a dataframe. GapEncoder is a type of single-column transformer. Unlike most scikit-learn estimators, its fit, transform and fit_transform methods expect a single column (a pandas or polars Series) rather than a full dataframe. To apply this transformer to one or more columns in a dataframe, use it as a parameter in a skrub.TableVectorizer or sklearn.compose.ColumnTransformer.

Versions

System:
    python: 3.12.3 | packaged by conda-forge | (main, Apr 15 2024, 18:35:20) [Clang 16.0.6 ]
executable: /Users/vincentmaladiere/miniforge3/envs/sandbox/bin/python3.12
   machine: macOS-14.0-arm64-arm-64bit

Python dependencies:
      sklearn: 1.5.0
          pip: 24.0
   setuptools: 70.0.0
        numpy: 1.26.4
        scipy: 1.13.1
       Cython: None
       pandas: 2.2.2
   matplotlib: 3.9.0
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 8
         prefix: libopenblas
       filepath: /Users/vincentmaladiere/miniforge3/envs/sandbox/lib/python3.12/site-packages/numpy/.dylibs/libopenblas64_.0.dylib
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: armv8

       user_api: blas
   internal_api: openblas
    num_threads: 8
         prefix: libopenblas
       filepath: /Users/vincentmaladiere/miniforge3/envs/sandbox/lib/python3.12/site-packages/scipy/.dylibs/libopenblas.0.dylib
        version: 0.3.27
threading_layer: pthreads
   architecture: neoversen1

       user_api: openmp
   internal_api: openmp
    num_threads: 8
         prefix: libomp
       filepath: /Users/vincentmaladiere/miniforge3/envs/sandbox/lib/python3.12/site-packages/sklearn/.dylibs/libomp.dylib
        version: None
0.2.0.dev0

[jeromedockes]

that's expected if you pass them a dataframe (several columns in the ColumnTransformer). They will work if you pass a singlecolumn (GapEncoder(), "column_name") -- as scikit-learn's TfidfVectorizer for example.

that was a deliberate decision we made with @GaelVaroquaux . our reasoning is that with the TableVectorizer and the upcoming pipeline builder, skrub users are not expected to need the ColumnTransformer anymore.

the equivalent pipeline would look something like

from skrub.datasets import fetch_employee_salaries
from skrub import GapEncoder
from skrub import PipeBuilder
from skrub import selectors as s

bunch = fetch_employee_salaries()

transformer = PipeBuilder().apply(GapEncoder(), cols=s.string()).get_pipeline()
transformer.fit_transform(bunch.X)

or for this dataset you may want the TableVectorizer because the GapEncoder with default parameters might not work on the "gender" column I think so the cardinality threshold could be useful. With the skrub pipeline you could also do somehting like cols=s.string() - 'gender'

[GaelVaroquaux]

Maybe we should improve our error behavior to help the user understand that this is expected and the skrub tools should be used

[jeromedockes]

I can expand the error message to say that when used in the ColumnTransformer, the corresponding columns should be a single column

[Vincent-Maladiere]

Yes, right now the error message is plain wrong, as you can't use the GapEncoder with multiple columns in the ColumnTransformer.