side effect of CFUnflattener in more matured level

[TakahiroHaruyama]

Hi,

I'm modifying HexRaysDeob to work for other binary sample.
The sample uses opaque predicates condition when assigning values to block comparison variable.
For example, in the following flattened blocks, the next comparison magic value is always 0xC504A26C (highlighted).

So I'd like to apply the un-flattening function in more matured level like MMAT_GLBOPT2, not MMAT_LOCOPT.
I noticed the mapping between the variables and Hex-Rays mblock_t numbers is lost in more matured level due to the optimization, so different algorithm is needed for the mapping. But is there any other side effect in the more matured levels?
I'd like to know why you decided to implement in MMAT_LOCOPT.

Thanks.

[RolfRolles]

I targeted MMAT_LOCOPT for unflattening because of the obfuscation described under the "Odd Stack Manipulations" heading of my blog entry. The odd stack manipulations happen before the call analysis (finalized by MMAT_CALLS, the next phase after MMAT_LOCOPT) can figure out which writes to the stack correspond to arguments for a given function call. Thus, I wanted the blocks with the odd stack manipulations to be fused with the blocks that performed the calls, so that call analysis would succeed, and hence I performed unflattening at MMAT_LOCOPT.

Your situation might call for something different, in which case you could use a later maturity level, which might allow you to take advantage of more sophisticated aspects of the Hex-Rays API than I was able to. You might have to use a multi-phase analysis, like I did for this same obfuscation. (I extract arguments to __alloca_probe() at MMAT_LVARS, and then modify the database so as to facilitate analysis in earlier microcode phases.)

I noticed the mapping between the variables and Hex-Rays mblock_t numbers is lost in more matured level due to the optimization, so different algorithm is needed for the mapping. But is there any other side effect in the more matured levels?

I bet the Hex-Rays developers would struggle to answer that question concisely, let alone a third-party developer like me. I can't answer this question.

[TakahiroHaruyama]

Thanks Rolf! I'll consider the code in other maturity level.

BTW, I could de-obfuscate opaque predicates of the sample by defining new patterns. Thank you so much for sharing your code.