FileNotFoundError when providing entry_point

[rogeriobromfman]

I can successfully deploy a HF model to Sagemaker using the standard inference methods. However, I can't get the custom inference to deploy. My code is the following:

from sagemaker.huggingface import HuggingFaceModel
import sagemaker

role = sagemaker.get_execution_role()

huggingface_model = HuggingFaceModel(
    model_data="s3://beaver-models-us/paragraph_classifier.tar.gz",   
    role=role,
    entry_point="inference.py",
    source_dir="./code",
    transformers_version='4.6.1',
    pytorch_version='1.7.1',
    py_version='py36',
)

predictor = huggingface_model.deploy(
    endpoint_name="paragraph-classifier",
    initial_instance_count=1,
    instance_type='ml.m5.xlarge'
)

The directory structure of paragraph_classifier.tar.gz is the following:

.
└── paragraph_classifier.tar.gz/
    ├── code/
    │   └── inference.py
    ├── pytorch_model.bin
    ├── config.json
    ├── tokenizer.json
    └── vocab.txt

When I run this, I get an error on the deploy() command saying that the code directory was not found! Full error stack below:

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-15-44909ec8a9da> in <module>
     18     endpoint_name="paragraph-classifier",
     19     initial_instance_count=1, # number of instances
---> 20     instance_type='ml.m5.xlarge' # ec2 instance type
     21 )

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/model.py in deploy(self, initial_instance_count, instance_type, serializer, deserializer, accelerator_type, endpoint_name, tags, kms_key, wait, data_capture_config, **kwargs)
    765                 self._base_name = "-".join((self._base_name, compiled_model_suffix))
    766 
--> 767         self._create_sagemaker_model(instance_type, accelerator_type, tags)
    768         production_variant = sagemaker.production_variant(
    769             self.name, instance_type, initial_instance_count, accelerator_type=accelerator_type

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/model.py in _create_sagemaker_model(self, instance_type, accelerator_type, tags)
    267                 /api/latest/reference/services/sagemaker.html#SageMaker.Client.add_tags
    268         """
--> 269         container_def = self.prepare_container_def(instance_type, accelerator_type=accelerator_type)
    270 
    271         self._ensure_base_name_if_needed(container_def["Image"])

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/huggingface/model.py in prepare_container_def(self, instance_type, accelerator_type)
    272 
    273         deploy_key_prefix = model_code_key_prefix(self.key_prefix, self.name, deploy_image)
--> 274         self._upload_code(deploy_key_prefix, repack=True)
    275         deploy_env = dict(self.env)
    276         deploy_env.update(self._framework_env_vars())

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/model.py in _upload_code(self, key_prefix, repack)
   1144                 repacked_model_uri=repacked_model_data,
   1145                 sagemaker_session=self.sagemaker_session,
-> 1146                 kms_key=self.model_kms_key,
   1147             )
   1148 

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/utils.py in repack_model(inference_script, source_directory, dependencies, model_uri, repacked_model_uri, sagemaker_session, kms_key)
    413 
    414         _create_or_update_code_dir(
--> 415             model_dir, inference_script, source_directory, dependencies, sagemaker_session, tmp
    416         )
    417 

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/utils.py in _create_or_update_code_dir(model_dir, inference_script, source_directory, dependencies, sagemaker_session, tmp)
    456         if os.path.exists(code_dir):
    457             shutil.rmtree(code_dir)
--> 458         shutil.copytree(source_directory, code_dir)
    459     else:
    460         if not os.path.exists(code_dir):

~/anaconda3/envs/python3/lib/python3.6/shutil.py in copytree(src, dst, symlinks, ignore, copy_function, ignore_dangling_symlinks)
    313 
    314     """
--> 315     names = os.listdir(src)
    316     if ignore is not None:
    317         ignored_names = ignore(src, names)

FileNotFoundError: [Errno 2] No such file or directory: './code'

What am I doing wrong here? Has anyone successfully deployed with custom inference code?

Thank you!!

[marshmellow77]

Hi Rogerio, there is two ways to tell the inference toolkit to use a custom inference script:

• You can have the inference script in your local environment and then point to it as you did when you created the HuggingFaceModel() instance
• Or you can have the inference code in the tar.gz file, in which case there is no need to specify where the inference script is located.

In our call I only mentioned the first option, but in the links I sent to you they only mention the second option, so this is totally my fault, sorry. But when you use both options you get the error message you are seeing.

I personally have used the first approach in this notebook which I tested and it works: https://github.com/marshmellow77/text-summarisation-project/blob/main/4a_model_testing_deployed.ipynb

Hope this helps!

Cheers, Heiko

[philschmid]

Hey @rogeriobromfman,

The error No such file or directory: './code' is showing up since you have defined source_dir="./code", in your HuggingFaceModel.
When you define source_dir & entrypoint in your HuggingFaceModel the python-sagemaker-sdk is looking for those file on the machine where you execute HuggingFaceModel.deploy and not inside the model.tar.gz.
That way there is no requirement to bundle your inference.py into model.tar.gz.

But if you have bundled the inference.py in your model.tar.gz you can simply remove source_dir and entry_point from the HuggingFaceModel and it should work.

[rogeriobromfman]

It worked! Having the inference code in the Notebook instance instead of the tar.gz file is what I was missing. Thank you so much Heiko and Philipp!!