initializing codeblock object slow

Question

initializing codeblock object slow

Closed this issue 8 years ago · 27 comments

I am iterating through a list of addresses and colors for functions, lines, and codeblocks. I use identical code for each with the only difference being whether sark.Function(address)/sark.Line(address) is called or sark.CodeBlock(address). The issue is that when called on my test files the line and function loops take only .05 seconds. The loop that calls codeblock takes 6.5 and has dramatically decreased the speed of my program.

Answer 1 · 2016-06-29T11:20:36.000Z

This is due to the way a codeblock is generated. It currently needs to:

Get a flowchart for the containing function
Iterate over the flowchart to find the block

And doing this for every address takes a while. I will look into that and see what I can do.

Answer 2 · 2016-06-29T14:33:31.000Z

I suspected that it was something like that. Thanks for getting back so
quickly, and thank you for creating this tool. It has certainly made my
life easier.

On Wed, Jun 29, 2016 at 7:20 AM, Tamir Bahar notifications@github.com
wrote:

This is due to the way a codeblock is generated. It currently needs to:

Get a flowchart for the containing function

Iterate over the flowchart to find the block

And doing this for every address takes a while. I will look into that and
see what I can do.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#44 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AIkBQCqe61641TIoImyrpF787FSQ65HHks5qQlUFgaJpZM4JAMpH
.

Answer 3 · 2016-06-29T15:23:28.000Z

Thanks for the feedback 😄

Coming up with a proper solution might take me a while (as I don't like the idea of caching results, and am not sure that there is an alternative here), but you can probably find a simple solution for your needs.

My approach (not knowing exactly what your requirements are) would be to iterate over all functions using sark.functions(), get the flowcharts using sark.FlowChart(function), and then iterating the blocks using for block in flowchart. This should (hopefully) be a lot faster. I hope this suits your needs.

Answer 4 · 2016-06-29T16:15:13.000Z

All right, I will implement that and let you know. Thanks.

On Wed, Jun 29, 2016 at 11:23 AM, Tamir Bahar notifications@github.com
wrote:

Thanks for the feedback 😄

Coming up with a proper solution might take me a while (as I don't like
the idea of caching results, and am not sure that there is an alternative
here), but you can probably find a simple solution for your needs.

My approach (not knowing exactly what your requirements are) would be to
iterate over all functions using sark.functions(), get the flowcharts
using sark.FlowChart(function), and then iterating the blocks using for
block in flowchart. This should (hopefully) be a lot faster. I hope this
suits your needs.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#44 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AIkBQAa5FADlM2Viog3TWiUsrTp49SbOks5qQo3xgaJpZM4JAMpH
.

Answer 5 · 2016-06-29T17:12:30.000Z

So I took your advice and used this

def gen_block_dict(self):
block_dict = {}
for func in sark.functions():
flow_chart = sark.FlowChart(func.ea)
for block in flow_chart:
block_dict[block.startEA] = block
return block_dict

which decreased the time it took by a factor of 50!

previously I had called

sark.CodeBlock(EA)

which was modified to

b_dict = self.gen_block_dict()

sblock = b_dict[EA]

Anyway, may I suggest that you add a blocks() function similar to the
lines() and function() functions?

Thanks
Fraser Hood

On Wed, Jun 29, 2016 at 12:15 PM, Fraser Hood fraserhood@gmail.com wrote:

All right, I will implement that and let you know. Thanks.

On Wed, Jun 29, 2016 at 11:23 AM, Tamir Bahar notifications@github.com
wrote:

Thanks for the feedback 😄

Coming up with a proper solution might take me a while (as I don't like
the idea of caching results, and am not sure that there is an alternative
here), but you can probably find a simple solution for your needs.

My approach (not knowing exactly what your requirements are) would be to
iterate over all functions using sark.functions(), get the flowcharts
using sark.FlowChart(function), and then iterating the blocks using for
block in flowchart. This should (hopefully) be a lot faster. I hope this
suits your needs.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#44 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AIkBQAa5FADlM2Viog3TWiUsrTp49SbOks5qQo3xgaJpZM4JAMpH
.

Answer 6 · 2016-06-30T09:53:22.000Z

I began to implement the function, and found a much better solution.

Just use sark.FlowChart(bounds=(start, end)). It gives a flowchart with every block in the range. Fast and easy. It will also include non-function blocks, so be aware of that.

This is actually really cool, I'm happy to have come across it.

I will add it to Sark, though.

Answer 7 · 2016-06-30T12:07:56.000Z

Added the function in https://github.com/tmr232/Sark/tree/codeblocks, let me know if it works for you.

Answer 8 · 2016-06-30T14:09:45.000Z

thanks

On Thu, Jun 30, 2016 at 8:07 AM, Tamir Bahar notifications@github.com
wrote:

Added the function in https://github.com/tmr232/Sark/tree/codeblocks, let
me know if it works for you.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#44 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AIkBQKIaEOvP7Dg5Z4CYpIwJmuODkyvPks5qQ7GdgaJpZM4JAMpH
.

Answer 9 · 2016-06-30T15:58:31.000Z

So i implemented it like so:

for block in sark.codeblocks():
print block.color
block_dict[block.startEA] = block

and what I found is that the block.color is always None, but it picks up
the line colors

for block in sark.codeblocks():
print block.color
block_dict[block.startEA] = block
for line in block.lines:
print line.color
block_dict[line.ea] = block

just fine. Let me know if I am implementing this wrong.

On Thu, Jun 30, 2016 at 10:09 AM, Fraser Hood fraserhood@gmail.com wrote:

thanks

On Thu, Jun 30, 2016 at 8:07 AM, Tamir Bahar notifications@github.com
wrote:

Added the function in https://github.com/tmr232/Sark/tree/codeblocks,
let me know if it works for you.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#44 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AIkBQKIaEOvP7Dg5Z4CYpIwJmuODkyvPks5qQ7GdgaJpZM4JAMpH
.

Answer 10 · 2016-06-30T16:07:01.000Z

Block color and line color are two different things. You can see the implementation here - https://github.com/tmr232/Sark/blob/master/sark/codeblocks.py#L44

What are you trying to do?

Answer 11 · 2016-06-30T16:13:52.000Z

No I realize that. The issue is that one of my blocks is colored, and yet
the print block.color statement never prints anything other than None.

On Thu, Jun 30, 2016 at 12:07 PM, Tamir Bahar notifications@github.com
wrote:

Block color and line color are two different things. You can see the
implementation here -
https://github.com/tmr232/Sark/blob/master/sark/codeblocks.py#L44

What are you trying to do?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#44 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AIkBQNifTj4nDUf9W7OODg4Zv6-AiySgks5qQ-mmgaJpZM4JAMpH
.

Answer 12 · 2016-06-30T16:14:57.000Z

I brought up the lines because they are printing the color as I would
expect them to.

On Thu, Jun 30, 2016 at 12:13 PM, Fraser Hood fraserhood@gmail.com wrote:

No I realize that. The issue is that one of my blocks is colored, and yet
the print block.color statement never prints anything other than None.

On Thu, Jun 30, 2016 at 12:07 PM, Tamir Bahar notifications@github.com
wrote:

Block color and line color are two different things. You can see the
implementation here -
https://github.com/tmr232/Sark/blob/master/sark/codeblocks.py#L44

What are you trying to do?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#44 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AIkBQNifTj4nDUf9W7OODg4Zv6-AiySgks5qQ-mmgaJpZM4JAMpH
.

Answer 13 · 2016-06-30T16:17:04.000Z

So to clarify, this

for func in sark.functions():
flow_chart = sark.FlowChart(func.ea)
for block in flow_chart:
print block.color

prints out the color that I expect, where as

for block in sark.codeblocks():
print block.color

this does not.

On Thu, Jun 30, 2016 at 12:14 PM, Fraser Hood fraserhood@gmail.com wrote:

I brought up the lines because they are printing the color as I would
expect them to.

On Thu, Jun 30, 2016 at 12:13 PM, Fraser Hood fraserhood@gmail.com
wrote:

No I realize that. The issue is that one of my blocks is colored, and yet
the print block.color statement never prints anything other than None.

On Thu, Jun 30, 2016 at 12:07 PM, Tamir Bahar notifications@github.com
wrote:

Block color and line color are two different things. You can see the
implementation here -
https://github.com/tmr232/Sark/blob/master/sark/codeblocks.py#L44

What are you trying to do?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#44 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AIkBQNifTj4nDUf9W7OODg4Zv6-AiySgks5qQ-mmgaJpZM4JAMpH
.

Answer 14 · 2016-06-30T18:40:09.000Z

Is the color of the block set in the graph view? If it is, it sounds like a bug in the color extraction.

Answer 15 · 2016-06-30T19:10:36.000Z

Yes it is.

On Thu, Jun 30, 2016 at 2:40 PM, Tamir Bahar notifications@github.com
wrote:

Is the color of the block set in the graph view? If it is, it sounds like
a bug in the color extraction.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#44 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AIkBQMNvaTb7E6uWU7s2JHtQxWcuy1ljks5qRA2KgaJpZM4JAMpH
.

Answer 16 · 2016-06-30T19:26:02.000Z

Can you show a screenshot of the graph overview?
And which version of IDA are you using?

Answer 17 · 2016-07-05T13:28:21.000Z

Sorry to bother you, but I was wondering if there was any news on the issue.

On Thu, Jun 30, 2016 at 3:53 PM, Fraser Hood fraserhood@gmail.com wrote:

On Thu, Jun 30, 2016 at 3:26 PM, Tamir Bahar notifications@github.com
wrote:

Can you show a screenshot of the graph overview?
And which version of IDA are you using?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#44 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AIkBQPs8tOoG6D1jbreYJ4JReH50n0G5ks5qRBhLgaJpZM4JAMpH
.

Answer 18 · 2016-07-06T13:25:35.000Z

OK, solved it.
You need to use the function's flowchart, and not just any chart. Should work now. I wonder how it affects performance.

Answer 19 · 2016-07-06T13:54:26.000Z

I will let you know.

On Wed, Jul 6, 2016 at 9:25 AM, Tamir Bahar notifications@github.com
wrote:

OK, solved it.
You need to use the function's flowchart, and not just any chart. Should
work now. I wonder how it affects performance.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#44 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AIkBQDRjUhEZ59R3eIuNyJEK2l38bJJfks5qS6zQgaJpZM4JAMpH
.

Answer 20 · 2016-07-06T14:59:01.000Z

So I am still having a similar issue albeit not exactly the same. The
codeblock colors are not updating properly but the line colors are. That
is, if I set a codeblock to lime green (#00ff00) save that information and
then set it back to white, when I try to change it back to green from the
saved info, it changes the line colors (is what it looks like) to green but
the codeblock stays as white. Ill include some pics to illustrate.

On Wed, Jul 6, 2016 at 9:54 AM, Fraser Hood fraserhood@gmail.com wrote:

I will let you know.

On Wed, Jul 6, 2016 at 9:25 AM, Tamir Bahar notifications@github.com
wrote:

OK, solved it.
You need to use the function's flowchart, and not just any chart. Should
work now. I wonder how it affects performance.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#44 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AIkBQDRjUhEZ59R3eIuNyJEK2l38bJJfks5qS6zQgaJpZM4JAMpH
.

Answer 21 · 2016-07-06T15:19:33.000Z

Can you also say what version of IDA are you using? And on which OS?

Answer 22 · 2016-07-06T16:12:42.000Z

Version 6.9.160222 (64-bit)
Windows 10

On Wed, Jul 6, 2016 at 11:19 AM, Tamir Bahar notifications@github.com
wrote:

Can you also say what version of IDA are you using? And on which OS?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#44 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AIkBQPHIj_7TMHfb4RrtC1d8E0jZXKvlks5qS8eGgaJpZM4JAMpH
.

Answer 23 · 2016-07-07T06:56:11.000Z

Same for me, and the new blocks code seems to work. Weird.

Answer 24 · 2016-07-07T13:36:54.000Z

Yeah I noticed that it seemed like it was retrieving the colors correctly
when I printed it out.

On Thu, Jul 7, 2016 at 2:56 AM, Tamir Bahar notifications@github.com
wrote:

Same for me, and the new blocks code seems to work. Weird.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#44 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AIkBQFtB3HPFOO4ELr4n34ZV5MMIk0y_ks5qTKMMgaJpZM4JAMpH
.

Answer 25 · 2016-07-13T12:03:31.000Z

So are there any remaining issues, or can I consider it solved?

Answer 26 · 2016-07-14T13:29:22.000Z

Ill let you know. Been really busy.

On Wed, Jul 13, 2016 at 8:03 AM, Tamir Bahar notifications@github.com
wrote:

So are there any remaining issues, or can I consider it solved?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#44 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AIkBQLLGqFfSzgp4nd6wMg0dUNdWF0ADks5qVNQUgaJpZM4JAMpH
.

Answer 27 · 2016-07-15T16:29:04.000Z

Works great! Awesome!

On Thu, Jul 14, 2016 at 9:29 AM, Fraser Hood fraserhood@gmail.com wrote:

Ill let you know. Been really busy.

On Wed, Jul 13, 2016 at 8:03 AM, Tamir Bahar notifications@github.com
wrote:

So are there any remaining issues, or can I consider it solved?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#44 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AIkBQLLGqFfSzgp4nd6wMg0dUNdWF0ADks5qVNQUgaJpZM4JAMpH
.