Coverage not working for TensorFlow Model call function

[NathanDotTo]

This problem was first reported at https://youtrack.jetbrains.com/oauth?state=%2Fissue%2FPY-38366

1 - I am running a unit test that is invoking the call(...) function of a TensorFlow model. I can see the trace message, that I have placed in the function, printed so I know the function is being called.
2 - The coverage analysis shows that code in red, as though it was not invoked.
3 - It should show green as it was invoked.

See test_model.py in https://github.com/Data-Science-Projects/demo-routenet.

See also the requirements.txt in that project.

[nedbat]

I've never used TensorFlow before. I'm going to guess that the execution of Python code within call(...) doesn't trigger the Python trace function, but I don't know.

Can you provide very very explicit reproduction steps? Include the version of Python, the commands to run to install everything, and the commands to run the program. Thanks.

[NathanDotTo]

Python 3.7.4 on macOS 10.14.6.

The steps so far are:

git clone https://github.com/Data-Science-Projects/demo-routenet.git
cd demo-routenet/bin
. ./create_routenet_venv.sh
cd ../tests/unit
pytest -s test_model.py --cov

Since I am looking the results in PyCharm, I am not quite sure how to recreate what I see in PyCharm at the CLI. What seems to be missing, in PyCharm, is any indication that the call function in the src/routenet/model/routenet_model.py code is covered. It is clear that the code is called, as one can see the output from the print statement.

[nedbat]

Thanks, this lets me reproduce the problem. I can see that line 65 in routenet_model.py is reported as uncovered. I'll dig into it.

[NathanDotTo]

Thanks, apologies for the delay in following up before. This is a weekend project ...

…

On 26 Oct 2019, at 13:18, Ned Batchelder ***@***.***> wrote: Thanks, this lets me reproduce the problem. I can see that line 65 in routenet_model.py is reported as uncovered. I'll dig into it. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#856?email_source=notifications&email_token=ADGFXFVWAHUQTMPGANGH4ULQQQRPPA5CNFSM4I7ZSAJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECKFXVI#issuecomment-546593749>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADGFXFSA3SMVWUJYOWW7CJLQQQRPPANCNFSM4I7ZSAJA>.

[nedbat]

@NathanDotTo this was an interesting one! It turns out that TensorFlow is making a transformed copy of your code, and then running it. So your original file isn't actually executed. In a way, coverage.py was right! I've opened tensorflow/tensorflow#33759 with TensorFlow to see what we can do about it.

[NathanDotTo]

Thank you. This may also change with TF2.0, which is changing from a graph, run in a session, programming model, to something more like traditional Python with “eager” processing. I am still coming to grips with TF2.0.

…

On 27 Oct 2019, at 13:28, Ned Batchelder ***@***.***> wrote: @NathanDotTo <https://github.com/NathanDotTo> this was an interesting one! It turns out that TensorFlow is making a transformed copy of your code, and then running it. So your original file isn't actually executed. In a way, coverage.py was right! I've opened tensorflow/tensorflow#33759 <tensorflow/tensorflow#33759> with TensorFlow to see what we can do about it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#856?email_source=notifications&email_token=ADGFXFUSA56JATJSNDS6JDTQQWCOBA5CNFSM4I7ZSAJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECK5GIQ#issuecomment-546689826>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADGFXFWGV5ATOW6KVSFNUSLQQWCOBANCNFSM4I7ZSAJA>.

[Zminghua]

I have a similar question, when I debug tensor2tensor and set a breakpoint in t2t_model.py at line 316, the debugger doesn't break as if there is no breakpoint. Then I run the code step by step, in the end, I also found a temporary Python file in /tmp/.

• tensorflow 1.15
• python 3.7

[nedbat]

@Zminghua debugging is not a coverage.py concern, but sharing my new-found expertise, it should work if you put a "pdb.set_trace()" line in your t2t_model.py file. That line will then be copied to the temporary file, and the debugger will break. You will be in a slightly strange world, since your file has been changed, but you can at least be in the debugger.

[Zminghua]

@nedbat Thanks for your reminding. I have learned that it is AutoGraph conversion in Tensorflow keras module. After I added a "pdb.set_trace()" line, I was really in a slightly strange world. However, I have modified tensorflow source code to prevent the conversion. Thank you very very very much.