[FEA] Make classifiers work with HuggingFace directly

Question

[FEA] Make classifiers work with HuggingFace directly

VibhuJawa opened this issue 5 months ago · 5 comments

With PR #58, we have cleaned up the model initialization quite a bit. We still need to make the classifiers work directly with HuggingFace without downloading weights.

We should do that once we upload the models to HF.

See comment here:
#58 (comment)

Answer 1 · 2024-05-22T01:01:22.000Z

Once this is done, we can also update https://github.com/NVIDIA/NeMo-Curator/blob/main/tutorials/distributed_data_classification/distributed_data_classification.ipynb and https://github.com/NVIDIA/NeMo-Curator/blob/main/docs/user-guide/DistributedDataClassification.rst accordingly.

Answer 2 · 2024-06-24T20:10:58.000Z

Can start working on this now that we have https://huggingface.co/nvidia/domain-classifier

Answer 3 · 2024-06-24T20:12:13.000Z

Yes please, @sarahyurick , Do you have band-width to start on this, happy to jump on it if not.

Answer 4 · 2024-06-24T20:17:49.000Z

Yes please, @sarahyurick , Do you have band-width to start on this, happy to jump on it if not.

Yes, happy to work on this.

Answer 5 · 2024-06-24T22:36:46.000Z

Linking to #71.

We should be able to get labels from HF directly (https://huggingface.co/nvidia/domain-classifier) as they are present in config.id2label.