Add support for IDA 7.4 & Python 3

Question

Add support for IDA 7.4 & Python 3

tmr232 opened this issue 5 years ago · 22 comments

Loading the plugin in IDA 7.4 Python 3 requires some minor tweaking:

Run 2to3
Add a None check to https://github.com/eset/ipyida/blob/master/ipyida/kernel.py#L29 as sys.__stdout__ is None for some reason.

Once done, the plugin loads and runs. However, it completely fails whenever any exception is thrown:

Jupyter QtConsole 4.5.1
Python 3.7.4 (tags/v3.7.4:e09359112e, Jul  8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.6.1 -- An enhanced Interactive Python. Type '?' for help.

few
ERROR:root:Internal Python error in the inspect module.
Below is the traceback from this internal error.

Traceback (most recent call last):
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\lib\site-packages\IPython\core\interactiveshell.py", line 3325, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-1-6e4eba6e7240>", line 1, in <module>
    few
NameError: name 'few' is not defined

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\lib\site-packages\IPython\core\interactiveshell.py", line 2039, in showtraceback
    stb = value._render_traceback_()
AttributeError: 'NameError' object has no attribute '_render_traceback_'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\lib\site-packages\IPython\core\ultratb.py", line 1101, in get_records
    return _fixed_getinnerframes(etb, number_of_lines_of_context, tb_offset)
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\lib\site-packages\IPython\core\ultratb.py", line 319, in wrapped
    return f(*args, **kwargs)
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\lib\site-packages\IPython\core\ultratb.py", line 353, in _fixed_getinnerframes
    records = fix_frame_records_filenames(inspect.getinnerframes(etb, context))
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\Lib\inspect.py", line 1502, in getinnerframes
    frameinfo = (tb.tb_frame,) + getframeinfo(tb, context)
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\Lib\inspect.py", line 1460, in getframeinfo
    filename = getsourcefile(frame) or getfile(frame)
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\Lib\inspect.py", line 696, in getsourcefile
    if getattr(getmodule(object, filename), '__loader__', None) is not None:
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\Lib\inspect.py", line 739, in getmodule
    f = getabsfile(module)
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\Lib\inspect.py", line 708, in getabsfile
    _filename = getsourcefile(object) or getfile(object)
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\Lib\inspect.py", line 684, in getsourcefile
    filename = getfile(object)
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\Lib\inspect.py", line 647, in getfile
    raise TypeError('{!r} is a built-in module'.format(object))
TypeError: <module '__plugins__plugin_loader' from ''> is a built-in module
---------------------------------------------------------------------------

Note that __plugins__plugin_loader is the name of my plugin loader (plugin_loader). If this is used without the plugin loader - the result is the same, but a different module name appears.

I believe this should probably be fixed in IPython, but thought it important to report here as well.

Answer 1 · 2019-10-07T01:51:34.000Z

Ho yes supporting 7.4 and Python 3 is a must. I'll ask for the beta now so I can test and update this week. I'm pretty such the switch to Py3 will break a few things...

There are also a few interesting things I left unreleased in the master branch (including somewhat-support for dark mode). I'll tackle this and do a release soon.

Answer 2 · 2019-10-07T09:40:37.000Z

Hi @marc-etienne, @tmr232 ,

(This is arnaud-at-hex-rays.com)

Although it is apparently not quite clear what the problem is at this point, please let me know if there's something to be done on IDA's side to support IPython, once you know what it is.
I cannot spend much time on IPython itself, but if there's something IDAPython does poorly which causes these failures, we're definitely willing to rectify that.

Answer 3 · 2019-10-07T14:33:23.000Z

Thanks @aundro !

I've requested the beta a few hours ago. Once I have a copy I'll test it out and update the status here.

Answer 4 · 2019-10-21T15:02:34.000Z

I started looking into this last Friday using the stable 7.4 macOS release. I didn't need to change anything in IPyIDA source to make it works (yay!).

However, I am able to reproduce the ERROR:root:Internal Python error in the inspect module [...] TypeError: <module '****' from ''> is a built-in module.

I'm still trying to find the root cause. Here are some observation:

It only happens on a NameError, try raising a syntax SyntaxError works as expected
Something is trying to call getfile() using a built-in module, but that seems to be in IPython
After 4 or 5 NameErrors, it goes back to normal and exceptions are properly displayed.
Each TypeError from getfile() shows a different module name (__main__, __plugins__ipyida, ...)

I will continue investigating.

Answer 5 · 2019-10-21T15:05:41.000Z

About this:

Add a None check to https://github.com/eset/ipyida/blob/master/ipyida/kernel.py#L29 as sys.stdout is None for some reason.

@tmr232 : Under what OS?

Answer 6 · 2019-10-21T15:56:13.000Z

I was able to reproduce without IPyIDA. Simply by doing the following:

Open IDA 7.4 (no need for opening a file)
In the console, type: import inspect; inspect.stack()

Output:

Python>import inspect
Python>inspect.stack()
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 1513, in stack
    return getouterframes(sys._getframe(1), context)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 1490, in getouterframes
    frameinfo = (frame,) + getframeinfo(frame, context)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 1460, in getframeinfo
    filename = getsourcefile(frame) or getfile(frame)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 696, in getsourcefile
    if getattr(getmodule(object, filename), '__loader__', None) is not None:
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 739, in getmodule
    f = getabsfile(module)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 708, in getabsfile
    _filename = getsourcefile(object) or getfile(object)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 684, in getsourcefile
    filename = getfile(object)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 647, in getfile
    raise TypeError('{!r} is a built-in module'.format(object))
TypeError: <module '__main__' (built-in)> is a built-in module
Python>inspect.stack()
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 1513, in stack
    return getouterframes(sys._getframe(1), context)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 1490, in getouterframes
    frameinfo = (frame,) + getframeinfo(frame, context)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 1460, in getframeinfo
    filename = getsourcefile(frame) or getfile(frame)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 696, in getsourcefile
    if getattr(getmodule(object, filename), '__loader__', None) is not None:
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 739, in getmodule
    f = getabsfile(module)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 708, in getabsfile
    _filename = getsourcefile(object) or getfile(object)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 684, in getsourcefile
    filename = getfile(object)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/inspect.py", line 647, in getfile
    raise TypeError('{!r} is a built-in module'.format(object))
TypeError: <module '__plugins__ipyida' from ''> is a built-in module
Python>inspect.stack()
Python>print(inspect.stack())
[FrameInfo(frame=<frame at 0x1186b5868, file '<string>', line 1, code <module>>, filename='<string>', lineno=1, function='<module>', code_context=None, index=None)]

I'm not sure if this is a could be a bug in Python's inspect.py or something funky IDAPython does.
@aundro : let me know if you think of anything.

Answer 7 · 2019-10-21T19:07:52.000Z

I found the root cause and was able to fix the issue with a single line change in IDAPython. @aundro expect a pull request very soon.

Answer 8 · 2019-10-22T07:41:55.000Z

Good to hear. Hopefully that single line change won't have unintended consequences on other scripts.

Answer 9 · 2019-10-22T11:39:52.000Z

This sounds like the same issue I was seeing with idaapi.require. Happy to hear a solution exists :)

Answer 10 · 2019-10-22T11:47:22.000Z

Running import inspect; inspect.stack() in a fresh new IDA 7.4 session works for me (both on windows & linux.)

I wonder what's causing the change in behavior on your box…

Answer 11 · 2019-10-22T12:09:25.000Z

Works on a fresh install, but seems to fail if I have any Python plugins loaded.

Answer 12 · 2019-10-22T13:04:30.000Z

I just:

added the SDK's idasdk74/plugins/script_plg/pyplugin.py to my IDA 7.4 install,
started IDA, and ran the plugin

-> import inspect; inspect.stack() still works for me.

Answer 13 · 2019-10-22T13:27:41.000Z

I think the plugin flags have to be ida_idaapi.PLUGIN_FIX for reproduction.

This is what I get:

Python>import inspect; inspect.stack()
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\Lib\inspect.py", line 1513, in stack
    return getouterframes(sys._getframe(1), context)
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\Lib\inspect.py", line 1490, in getouterframes
    frameinfo = (frame,) + getframeinfo(frame, context)
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\Lib\inspect.py", line 1460, in getframeinfo
    filename = getsourcefile(frame) or getfile(frame)
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\Lib\inspect.py", line 696, in getsourcefile
    if getattr(getmodule(object, filename), '__loader__', None) is not None:
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\Lib\inspect.py", line 739, in getmodule
    f = getabsfile(module)
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\Lib\inspect.py", line 708, in getabsfile
    _filename = getsourcefile(object) or getfile(object)
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\Lib\inspect.py", line 684, in getsourcefile
    filename = getfile(object)
  File "C:\Users\tamir.bahar\AppData\Local\Programs\Python\Python37\Lib\inspect.py", line 647, in getfile
    raise TypeError('{!r} is a built-in module'.format(object))
TypeError: <module '__plugins__pyplugin' from ''> is a built-in module

Answer 14 · 2019-10-22T14:25:51.000Z

@tmr232 ida_idaapi.PLUGIN_FIX with the example pyplugin.py, doesn't change anything for me.

I'd really like to be able to reproduce this, before I accept the PR. Can one of you guys put me on the right track?
If I understand @marc-etienne 's fix correctly, IDAPython_ExecScript could be called with globals coming from a builtin module. How can I reproduce this?

Answer 15 · 2019-10-22T14:40:52.000Z

@marc-etienne Using your patch I no longer get the exception, but IPython dies anyhow. I get a regular exception the first time, then every line results in:
ERROR: execution aborted

Answer 16 · 2019-10-22T14:48:34.000Z

@tmr232: Try installing IPyIDA from the master branch

Answer 17 · 2019-10-22T15:10:08.000Z

Sorry for the cliffhanger yesterday :D. Basically, having __file__ set to an empty string is a edge case not handled properly in inspect.py. After IDAPython_ExecScript is executed, sys.modules["__main__"].__file__ (or a __plugin_XXX namespace) may end up being empty string. This will stay persistant. In inspect.py:getmodule(), the code will reach line 739 which iterates over all modules and enventually call getfile() using the built-in module, which raises a TypeError.

Yeah this is hard to follow :P, but seems like not tempering with __file__ too much avoid this error.

@aundro : can you tell me which Python version you have installed?

Answer 18 · 2019-10-22T15:17:33.000Z

Windows: 3.7.4
Linux: 3.5.3

Answer 19 · 2019-10-23T12:04:44.000Z

@marc-etienne Tested with latest, and it works.
Seems like my issue was having a newer version of IPython.

Now that we're using Python3, do you have any idea what it'd take to migrate to latest IPython/Jupyter?

Answer 20 · 2019-10-23T19:31:00.000Z

I added some code that works around the issue without having to modify IDAPython, so it should work out of the box on current IDA 7.4.

I will test on other platforms (more specifically on Windows), test if the install script needs changes too and do a release next week.

Seems like my issue was having a newer version of IPython.
Now that we're using Python3, do you have any idea what it'd take to migrate to latest IPython/Jupyter?

I also tested upgrading ipykernel to version >= 5 and I have the same ERROR: execution aborted error after a few successful commands. I tried finding the cause of that error for a few hours and I hasn't successful so far 😞.

Since this is not required for IDA 7.4 and Python 3 support, I will track this in #26.

Answer 21 · 2019-11-04T16:24:35.000Z

v1.4 is out, 🍾

Answer 22 · 2020-11-12T20:56:06.000Z

FYI: The fix for this breaks IDA 6.xx, where IDAPython_ExecScript takes only two arguments.