Error When Fine Tuning

[scblaze]

Thank you for creating this. Inference works fine for me, however when attempting to Fine Tune using the Colab version, I get this error:

The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization. 
The tokenizer class you load from this checkpoint is 'LLaMATokenizer'. 
The class this function is called from is 'LlamaTokenizer'.
trainable params: 4194304 || all params: 6742609920 || trainable%: 0.06220594176090199
RuntimeError: module compiled against API version 0x10 but this version of numpy is 0xf
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/transformers/utils/import_utils.py", line 1125, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/dist-packages/transformers/trainer.py", line 68, in <module>
    from .data.data_collator import DataCollator, DataCollatorWithPadding, default_data_collator
  File "/usr/local/lib/python3.9/dist-packages/transformers/data/__init__.py", line 26, in <module>
    from .metrics import glue_compute_metrics, xnli_compute_metrics
  File "/usr/local/lib/python3.9/dist-packages/transformers/data/metrics/__init__.py", line 18, in <module>
    if is_sklearn_available():
  File "/usr/local/lib/python3.9/dist-packages/transformers/utils/import_utils.py", line 558, in is_sklearn_available
    return is_scipy_available() and importlib.util.find_spec("sklearn.metrics")
  File "/usr/lib/python3.9/importlib/util.py", line 94, in find_spec
    parent = __import__(parent_name, fromlist=['__path__'])
  File "/usr/local/lib/python3.9/dist-packages/sklearn/__init__.py", line 82, in <module>
    from .base import clone
  File "/usr/local/lib/python3.9/dist-packages/sklearn/base.py", line 17, in <module>
    from .utils import _IS_32BIT
  File "/usr/local/lib/python3.9/dist-packages/sklearn/utils/__init__.py", line 25, in <module>
    from .fixes import parse_version, threadpool_info
  File "/usr/local/lib/python3.9/dist-packages/sklearn/utils/fixes.py", line 19, in <module>
    import scipy.stats
  File "/usr/local/lib/python3.9/dist-packages/scipy/stats/__init__.py", line 485, in <module>
    from ._stats_py import *
  File "/usr/local/lib/python3.9/dist-packages/scipy/stats/_stats_py.py", line 37, in <module>
    from numpy.testing import suppress_warnings
  File "/usr/local/lib/python3.9/dist-packages/numpy/testing/__init__.py", line 10, in <module>
    from ._private.utils import *
  File "/usr/local/lib/python3.9/dist-packages/numpy/testing/_private/utils.py", line 23, in <module>
    import numpy.linalg.lapack_lite
ImportError: numpy.core.multiarray failed to import

[zetavg]

Thanks for reporting. I think it's an issue caused by Colab preloading a older version of numpy, which is not reloaded after pip install -r requirements.lock.txt. Can you add the following code into the "A small workaround" code block, re-run and see if it'll fix the issue?

!pip install numpy==1.23.5
import numpy
major, minor = map(float, numpy.__version__.split(".")[:2])
version_float = major + minor / 10**len(str(minor))
print('numpy', version_float)
if version_float < 1.0023:
    raise Exception("Restart the runtime by clicking the 'RESTART RUNTIME' button above (or Runtime > Restart Runtime).")

[scblaze]

Thank you, that fixed the issue.

As an aside, do you have more information or documentation about the options? There are a lot of different options when fine tuning and it would be helpful to have more documentation, examples and best practices on how to use them.

[zetavg]

Yay, thanks for the update!

I tried it myself on Colab and reproduced the issue, and now the Colab Notebook has been updated to handle this.

For the options, while I'll add more detailed tooltips or notes about how they work after having a confident understanding of them by myself (don't want to introduce misrepresentation), I would recommend consulting the Hugging Face Transformers doc https://huggingface.co/docs/transformers/v4.28.0/en/main_classes/trainer#transformers.TrainingArguments about the parameters for training (still finding docs about LoRA related parameters...)