Endpoint creation completes before custom model_fn finishes loading resources

[Tripping-Hazard]

Image used: 763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-inference:2.1.0-transformers4.37.0-gpu-py310-cu118-ubuntu20.04 with custom inference.py.

I am loading some essential data from S3 for post processing on model's output and I included the data loading in the custom model_fn function. The data takes around 3~5 minutes to load. One thing I noticed is that the endpoint creation or update will complete before model_fn returns so the endpoint becomes available for incoming calls before all the data and model is loaded. This resulted in several minutes of additional latency around the period of time when endpoint is created or updated. How can I prevent this from happening?