how to load data in inference.py

[moshonkel]

Hey,

im implementing an endpoint that can perform semantic search on pre computed embeddings. I think everything is working fine, but i am having trouble loading data in the inference.py. How do i effectively do this (the embeddings and the csv i want to load are pretty small, so no need to use elastic search).

Do i put the data in the model.tar.gz, or load it seperately to s3? None of this seems to work.

import torch
import torch.nn.functional as F
import pandas as pd
import numpy as np

corpus_embeddings = np.load('s3://custommodels/sentence_embeddings.npy')
data = pd.read_csv('s3://custommodels/sentences.csv', index_col=0)

def similarity(embeddings_1, embeddings_2):
    normalized_embeddings_1 = F.normalize(embeddings_1, p=2)
    normalized_embeddings_2 = F.normalize(embeddings_2, p=2)
    return torch.matmul(
        normalized_embeddings_1, normalized_embeddings_2.transpose(0, 1)
    )

def output_fn(prediction, accept):
    prediction = torch.tensor(prediction)
    attention_mask = torch.ones(prediction.shape[1], dtype=torch.uint8)
    mask = attention_mask.unsqueeze(-1).expand(prediction.size()).float()
    masked_embeddings = prediction * mask
    summed = torch.sum(masked_embeddings, 1)
    summed_mask = torch.clamp(mask.sum(1), min=1e-9)
    query_embedding = (summed / summed_mask)
    scores = similarity(torch.tensor(corpus_embeddings, dtype=torch.float32), torch.tensor(query_embedding, dtype=torch.float32))
    maximum_score = torch.max(scores, 0)
    result = data.iloc[int(maximum_score[1])].to_dict()
    result['Ähnlichkeit'] = float(maximum_score[0][0])
    return result

and the error :

---------------------------------------------------------------------------
ModelError                                Traceback (most recent call last)
<ipython-input-53-590fc088c70c> in <module>
----> 1 test = predictor.predict({"inputs": "abcd"})
      2 
      3 
      4 test

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/sagemaker/predictor.py in predict(self, data, initial_args, target_model, target_variant, inference_id)
    159             data, initial_args, target_model, target_variant, inference_id
    160         )
--> 161         response = self.sagemaker_session.sagemaker_runtime_client.invoke_endpoint(**request_args)
    162         return self._handle_response(response)
    163 

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
    389                     "%s() only accepts keyword arguments." % py_operation_name)
    390             # The "self" in this scope is referring to the BaseClient.
--> 391             return self._make_api_call(operation_name, kwargs)
    392 
    393         _api_call.__name__ = str(py_operation_name)

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
    717             error_code = parsed_response.get("Error", {}).get("Code")
    718             error_class = self.exceptions.from_code(error_code)
--> 719             raise error_class(parsed_response, operation_name)
    720         else:
    721             return parsed_response

ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from primary with message "{
  "code": 400,
  "type": "InternalServerException",
  "message": "[Errno 2] No such file or directory: \u0027s3://custommodels/sentence_embeddings.npy\u0027"
}
". See https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logEventViewer:group=/aws/sagemaker/Endpoints/huggingface-pytorch-inference-2021-12-21-16-26-49-230 in account 736551082663 for more information.

[philschmid]

Hey @moshonkel,

what you described on what you want to sounds like a perfect use case for SageMaker Batch Transform, where you can provide files from s3 which are then used to run predictions.

I created a video and sample notebook on how to do batch transform: https://www.youtube.com/watch?v=lnTixz0tUBg&ab_channel=HuggingFace
To be able to use it you might need to create a custom inference.py with your custom logic.
Notebook: https://github.com/huggingface/notebooks/blob/master/sagemaker/12_batch_transform_inference/sagemaker-notebook.ipynb

[Deleted user]

Hey,

thanks for your answer. I do not understand how batch transform is helpful in this case.

I have an endpoint with a custom inference.py script, this is the inference.py:

import torch
import torch.nn.functional as F
import pandas as pd
import numpy as np

corpus_embeddings = np.load('s3://custommodels/sentence_embeddings.npy')
data = pd.read_csv('s3://custommodels/sentences.csv', index_col=0)

def similarity(embeddings_1, embeddings_2):
    normalized_embeddings_1 = F.normalize(embeddings_1, p=2)
    normalized_embeddings_2 = F.normalize(embeddings_2, p=2)
    return torch.matmul(
        normalized_embeddings_1, normalized_embeddings_2.transpose(0, 1)
    )

def output_fn(prediction, accept):
    prediction = torch.tensor(prediction)
    attention_mask = torch.ones(prediction.shape[1], dtype=torch.uint8)
    mask = attention_mask.unsqueeze(-1).expand(prediction.size()).float()
    masked_embeddings = prediction * mask
    summed = torch.sum(masked_embeddings, 1)
    summed_mask = torch.clamp(mask.sum(1), min=1e-9)
    query_embedding = (summed / summed_mask)
    scores = similarity(torch.tensor(corpus_embeddings, dtype=torch.float32), torch.tensor(query_embedding, dtype=torch.float32))
    maximum_score = torch.max(scores, 0)
    result = data.iloc[int(maximum_score[1])].to_dict()
    result['Ähnlichkeit'] = float(maximum_score[0][0])
    return result

The endpoint with the huggingfacemodel and the l2 regularization work fine the way i wrote it. Meaning i can get a correct sentence embedding from the endpoint.

What i am trying to do now is to compare the embedding from the inference of the endpoint, with embeddings, that are stored or deployed with the model

corpus_embeddings = np.load('s3://custommodels/sentence_embeddings.npy')

This does not work, and i dont understand how my inference.py can access data from anywhere (no matter if its on s3, or also deployed?, or anywhere else).
When i am in a sagemaker notebook, i can load files from s3 into ram. How is this done with an endpoint?

Ps the files i need are less than 10mb.

[Deleted user]

This is the solution i came up with. You can just load data from s3 in the inference.py . For my case this is the most practical solution.

import boto3
import io

s3 = boto3.client('s3')
obj = s3.get_object(Bucket='custommodels', Key='sentences.csv')
df = pd.read_csv(io.BytesIO(obj['Body'].read()))
sentences = pd.read_csv('s3://custommodels/sentences.csv', index_col=0)

Still, thanks for the help @philschmid

[b5y]

Hey @moshonkel !

The solution doesn't seem to be efficient. Here is why:

Every time when you call inference.py, you read the csv file from S3 bucket. It might be okay when the file is very small, but what will you do if the file is too large?

I am also facing similar problem right now and looking for the most efficient solution. Batch transform is also not something I would like to have as a proper solution.

Are there other ways to store the embeddings in the memory? Like we do with reading the model.

UPDATE: I do know there is a possibility to use some database, but currently want to avoid it.