wasm-opt -O2 seg faults on large binary with many nested if-then-else instructions

[mkarup]

I have a somewhat large binary (1.9 mb) that contains a function with a lot of nested if-then-else instructions, and running
wasm-opt -O2 <input_file.wasm> --enable-tail-call --enable-mutable-globals --output <output_file.wasm>
causes a segmentation fault.

The binary is almost exactly the same as the one described in this issue on wat2wasm. I can also provide more information if needed, and/ or the actual file.

Looking at the --debug output, it looks like the seg fault happens writing the binary to the output file, i.e. it looks like the optimization passes run successfully (I haven't yet debugged what the actual seg fault is).

What is strange is that running wasm-opt with only the --coalesce-locals optimization flag successfully applies the optimization and writes the output to the file.

[kripken]

The file from that linked issue shows the problem. It looks like a stack overflow,

#442 .. in wasm::WasmBinaryReader::visitIf(wasm::If*) () from lib/libbinaryen.so
#443 .. in wasm::WasmBinaryReader::readExpression(wasm::Expression*&) () from lib/libbinaryen.so
#444 .. in wasm::WasmBinaryReader::processExpressions() () from lib/libbinaryen.so
#445 .. in wasm::WasmBinaryReader::getBlockOrSingleton(wasm::Type) () from lib/libbinaryen.so
#446 .. in wasm::WasmBinaryReader::visitIf(wasm::If*) () from lib/libbinaryen.so

The same keeps repeating.

We already have logic to handle deeply-nested blocks, as that is a common pattern in switches. To fix this we'd need similar logic for ifs. However, if that's justified then we'd probably want to do something more general for all control flow structures, which would be a larger refactoring.

But it would be best to avoid this in your compiler, I think, as was suggested on the other issue.