Azure-functions with dill - failing on importing dill. No module named '__main__`

[Ceglowa]

Hi, I am trying to deploy a function to Azure that is using a dill library. Right now the function returns 500 immediately after calling. It fails when importing dill. What is interesting is that the same procedure with deploying the function worked around 3 weeks ago - and it was also using dill as a requirement. It shows the following error:

Result: Failure Exception: ModuleNotFoundError: No module named '__main__'. 
Please check the requirements.txt file for the missing module. For more info, please refer the troubleshooting guide: https://aka.ms/functions-modulenotfound    
Stack: 
File "/azure-functions-host/workers/python/3.7/LINUX/X64/azure_functions_worker/dispatcher.py", line 359, in _handle__function_load_request func_request.metadata.entry_point) 
File "/azure-functions-host/workers/python/3.7/LINUX/X64/azure_functions_worker/utils/wrappers.py", line 42, in call raise extend_exception_message(e, message) 
File "/azure-functions-host/workers/python/3.7/LINUX/X64/azure_functions_worker/utils/wrappers.py", line 40, in call return func(*args, **kwargs) 
File "/azure-functions-host/workers/python/3.7/LINUX/X64/azure_functions_worker/loader.py", line 127, in load_function mod = importlib.import_module(fullmodname) 
File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) 
File "/home/site/wwwroot/src/handler.py", line 4, in <module> import dill File "/home/site/wwwroot/.python_packages/lib/site-packages/dill/__init__.py", line 25, in <module> from ._dill import dump, dumps, load, loads, dump_session, load_session, \ 
File "/home/site/wwwroot/.python_packages/lib/site-packages/dill/_dill.py", line 80, in <module> import __main__ as _main_module

I've made sure that I am using the same version of azure-functions-core-tools as I was using when the deployment and call worked. I am happy to provide more details if necessary. Thank you in advance for response.

[mmckerns]

You'll need to provide more details, both in what "worked" previously, and what "doesn't" now (i.e. what versions of dill were/are you using, at the least). The only thing I can say at this point is that it's very unlikely that the line import __main__ as _main_module would cause a failure due to a change in dill. That particular line has been unchanged in the source code for 15 years. Three weeks ago, were you testing on the same system, using the same procedure? If not, is it possible that the target system doesn't have a __main__?

[mmckerns]

After a tiny bit of googling, this seems to be an azure issue and not a dill issue. See similar issue (and explanation and workaround) here: https://stackoverflow.com/questions/67904174/python-azure-function-failure-exception-modulenotfounderror-no-module-named. I'm going to close this issue, but please reopen it if indeed it isn't an issue with azure (configuration or otherwise).

If azure doesn't use __main__, then it might be worthwhile for dill to account for the import of __main__ failing. So, let me know below if you find that azure indeed doesn't use __main__.

[Ceglowa]

What was the procedure before when the function was working:

1. Deploy Function App via terraform
2. Install azure-functions-core-tools with version 4.0.4653-1using apt-get -y install azure-functions-core-tools-4=4.0.4653-1
3. Login to azure using az login
4. Switch to a specific subscription
5. Install python packages via pip install --target=".python_packages/lib/site-packages" -r requirements.txt - used dill version was 0.3.3
6. Deploy a function to azure using func azure functionapp publish app_name --python --no-build
7. Deployment ending with Deployment completed successfully.
8. Call to the function was working correctly. Regarding dill, my app doesn't have a __main__ and it was working back then.

What is the procedure right now (which is not working):

1. Deploy Function App via terraform - the configuration of the Function App has not changed
2. Install azure-functions-core-tools with version 4.0.4653-1using apt-get -y install azure-functions-core-tools-4=4.0.4653-1
3. Login to azure using az login
4. Switch to a specific subscription - subscription has not changed
5. Install python packages via pip install --target=".python_packages/lib/site-packages" -r requirements.txt - used dill version is 0.3.3 - I also tried removing everything from the function apart from dill and azure-functions library. With only those libraries the function also is not working properly.
6. Deploy a function to azure using func azure functionapp publish app_name --python --no-build
7. Deployment ending with Deployment completed successfully.
8. Call to the function is not working correctly. Regarding dill, my app still doesn't have a __main__ and right now it is not working.

The problem is interesting since you are stating that the __main__ should be there and back then it wasn't there and it was working. The code didn't change since then, there isn't any __main__ in my code and now it doesn't work.

I tried your workaround but for now no luck :/

[mmckerns]

It seems like you are using dill-0.3.3 in both cases... and it seems like your procedure hasn't changed. Is that correct? I'd have to assume that the azure configuration has changed, or something like that... otherwise, if you are doing everything the same and (apparently) using the same procedure/configuration, then it sounds like it should be a resource configuration change that's the issue. To me, it doesn't seem like a dill issue.

However, I'm going to reopen this, and investigate systems not having a __main__. Any resolution coming from this won't help you if you continue to use dill-0.3.3.

[leogama]

The problem is interesting since you are stating that the __main__ should be there and back then it wasn't there and it was working. The code didn't change since then, there isn't any __main__ in my code and now it doesn't work.

Just to clarify, @mmckerns is talking about "the" __main__ module, i.e. the top-level environment where code runs, which may or may not be the __main__.py file of a package (it depends on how the interpreter was invoked). From the docs:

__main__ is the name of the environment where top-level code is run. “Top-level code” is the first user-specified Python module that starts running. It’s “top-level” because it imports all other modules that the program needs. Sometimes “top-level code” is called an entry point to the application.

Python is supposed to have a __main__ module available independently of the environment or how it was initialized:

Regardless of which module a Python program was started with, other modules running within that same program can import the top-level environment’s scope (namespace) by importing the __main__ module. This doesn’t import a __main__.py file but rather whichever module that received the special name '__main__'.

Note that this is not an implementation detail of CPython, it's part of the language.

---

A simple workaround to make dill load is to create a mock __main__ module before importing it:

import sys
import types
sys.modules.setdefault('__main__', types.ModuleType('__main__'))
import dill  # or import the module that imports dill

[leogama]

@Ceglowa, does my suggested workaround works? I'm curious, as the situation you presented is very unusual.

[Ceglowa]

Hi, I didn't try out the suggested workaround. It turned out it was a problem with python runner that is used for azure functions. Here is an issue that says about this problem in azure-functions-python-worker github Azure/azure-functions-python-worker#1088. I'm closing this issue since it is a problem with the runner, not with dill. Current workaround is to rollback to the working runner version.